The company’s PaaS platform uses the open source kubespray for the underlying kubernetes cluster deployment, and I was involved in the kubespray second-opening work. During this period, I mainly completed kubespray automated packaging and release pipeline, private deployment, added self-research CNI deployment, and some bugfixes. I recently took time to organize and summarize some of the pitfalls of using kubespray to deploy kubernetes clusters for local development and testing.

Prepare

  • Need a deployment image repository and nginx
  • A domain name is required, preferably with DNS resolution and SSL certificates already set up
  • The cluster node needs to have at least two machines with access to the extranet

Although I have a large number of development machines on hand, I can’t resolve DNS to these Chinese servers because my domain name k8s.li is special and difficult to file in China (and I don’t want to file). So I plan to resolve the domain name to a foreign server, and then use nginx rewrite to forward the request to AliCloud OSS; in addition, the backend storage of docker registry can also choose to use AliCloud OSS, so that when the client pulls the image, it will only get the manifest file of the image through my domain name, and the image’s During cluster deployment, the most important traffic for downloading files and mirrors will go through AliCloud OSS, which can save cluster deployment time and improve deployment efficiency, while leaving a server traffic cost.

Domain Name SSL Certificate Creation

If the certificate is self-signed or the mirror repository uses the HTTP protocol, this will cause docker or containerd to be unable to pull mirrors and require the insecure-registries parameter to be configured for all nodes in the cluster. This is a bit of a pain, so it is recommended to add a non-self-signed SSL certificate to the mirror repository to reduce some of the unnecessary hassles. If you have a ready-made mirror repository with an SSL certificate configured, you can skip this step.

There are many ways to create domain certificates, but I recommend using acme.sh. It implements all the authentication protocols supported by the acme protocol, and supports dozens of domain name resolvers. As my domain is hosted on cloudflare, using acme.sh to issue certificates is particularly convenient, only two parameters need to be configured. The following to k8s.li this domain name issued a pan-domain certificate.

  • Install acme.sh

    1
2

    curl https://get.acme.sh | sh
~/.acme.sh/acme.sh --help

  • Issuing Certificate

     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    export CF_Email="muzi502.li@gmail.com" # cloudflare 账户的邮箱
export CF_Key="xxxxxx" # "cloudflare中查看你的key"

~/.acme.sh/acme.sh --issue --dns dns_cf -d k8s.li -d *.k8s.li

[Tue Apr 27 07:32:52 UTC 2021] Cert success.
[Tue Apr 27 07:32:52 UTC 2021] Your cert is in  /root/.acme.sh/k8s.li/k8s.li.cer
[Tue Apr 27 07:32:52 UTC 2021] Your cert key is in  /root/.acme.sh/k8s.li/k8s.li.key
[Tue Apr 27 07:32:52 UTC 2021] The intermediate CA cert is in  /root/.acme.sh/k8s.li/ca.cer
[Tue Apr 27 07:32:52 UTC 2021] And the full chain certs is there:  /root/.acme.sh/k8s.li/fullchain.cer

    After the previous certificate is generated, you need to copy the certificate to the place where you really need to use it.

    Note that the default generated certificates are placed in the installation directory ~/.acme.sh/, please do not use the files in this directory directly, e.g. do not let the nginx/apache configuration file use the files under it. The files in this directory are used internally and the directory structure may change.

    The correct way to use it is to use the -installcert command and specify the target location, then the certificate file will be copied to the appropriate location

  • Installation certificate

    1
2
3
4

    acme.sh --install-cert -d k8s.li \
--cert-file      /etc/nginx/ssl/k8s.li.cer  \
--key-file       /etc/nginx/ssl/k8s.li.key  \
--fullchain-file /etc/nginx/ssl/fullchain.cer

Build mirror repository

  • config.yml

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

    version: 0.1
log:
fields:
    service: registry
storage:
cache:
    blobdescriptor: inmemory
oss:
    accesskeyid: xxxx # 这里配置阿里云 OSS 的 accesskeyid
    accesskeysecret: xxxx # 这里配置阿里云 OSS 的 accesskeysecret
    region: oss-cn-beijing # 配置 OSS bucket 的区域，比如 oss-cn-beijing
    internal: false
    bucket: fileserver # 配置存储 bucket 的名称
    rootdirectory: /kubespray/registry # 配置路径
delete:
    enabled: true
http:
addr: 0.0.0.0:5000
headers:
    X-Content-Type-Options: [nosniff]
health:
storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

  • docker-compose

     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    version: '2'
services:
hub-registry:
    image: registry:2.7.1
    container_name: hub-registry
    restart: always
    volumes:
    - ./config.yml:/etc/docker/registry/config.yml
    ports:
    - 127.0.0.1:5000:5000

  • nginx.conf

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

    server {
    listen       443 ssl;
    listen       [::]:443;
    server_name  hub.k8s.li;
    ssl_certificate /etc/nginx/ssl/fullchain.cer;
    ssl_certificate_key /etc/nginx/ssl/k8s.li.key;
    gzip_static on;
    client_max_body_size 4096m;
    if ($request_method !~* GET) {
        return 403;
    }
    location / {
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass   http://localhost:5000;
    }
}

File Server

The file server is used to store some binaries such as kubeadm, kubectl, kubelet, etc. The default download address of kubespray is particularly slow in China, so an http/https server needs to be built to download these binaries for cluster deployments.

  • nginx.conf

    Note that the nginx configuration here uses rewrite instead of proxy_pass, so that when the client wants to request a file from my server, it will rewrite the client’s request and let the client request the AliCloud OSS address.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    server {
    listen 443;
    listen [::]:443;
    server_name   dl.k8s.li;
    ssl_certificate /etc/nginx/ssl/fullchain.cer;
    ssl_certificate_key /etc/nginx/ssl/k8s.li.key;
    location / {
        rewrite ^/(.*)$ https://fileserver.oss-cn-beijing.aliyuncs.com/kubespray/files/$1;
        proxy_hide_header Content-Disposition;
        proxy_hide_header x-oss-request-id;
        proxy_hide_header x-oss-object-type;
        proxy_hide_header x-oss-hash-crc64ecma;
        proxy_hide_header x-oss-storage-class;
        proxy_hide_header x-oss-force-download;
        proxy_hide_header x-oss-server-time;
    }
}

Compile and install skopeo

Installing skopeo to synchronize some used images to a private image repository is much faster than docker, and is highly recommended. skopeo installation can be found in the official documentation Installing from packages. However, I personally use go buid to compile a statically linked executable so that it can be used in all Linux distributions. Otherwise, the executable compiled on Debian cannot be used on CentOS because they use different dynamic link libraries!

1
2
3
4
5
6
7

root@debian:/root/skopeo git:(master*) # git clone https://github.com/containers/skopeo && cd skopeo

# 本地开发机器已经安装并配置好了 golang 编译环境
root@debian:/root/skopeo git:(master*) # CGO_ENABLE=0 GO111MODULE=on go build -mod=vendor "-buildmode=pie" -ldflags '-extldflags "-static"' -gcflags "" -tags "exclude_graphdriver_devicemapper exclude_graphdriver_btrfs containers_image_openpgp" -o bin/skopeo ./cmd/skopeo

root@debian:/root/skopeo git:(master*) # ldd bin/skopeo
not a dynamic executable

Get the binaries needed for deployment

When deploying kubespray, we need to download some binaries from github.com or storage.googleapis.com, which are blocked in China, so we need to upload the files we depend on to our own file server. I wrote a script to get the binaries needed for kubespray deployment, and executed it in the root directory of the kubespray repo, where the downloaded files are stored in the temp/files directory by default. After the download is complete, upload all subdirectories in this directory to your own file server. You can configure some parameters later by adding the URL of your file server in front of this address.

  • First clone repo to local

    1

    root@debian:/root# git clone https://github.com/kubernetes-sigs/kubespray && cd kubespray

  • Save the script generate_list.sh to the repo root directory and execute the footer to download the required files.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

    #!/bin/bash
set -eo pipefail

CURRENT_DIR=$(cd $(dirname $0); pwd)
TEMP_DIR="${CURRENT_DIR}/temp"
REPO_ROOT_DIR="${CURRENT_DIR}"

: ${IMAGE_ARCH:="amd64"}
: ${ANSIBLE_SYSTEM:="linux"}
: ${ANSIBLE_ARCHITECTURE:="x86_64"}
: ${DOWNLOAD_YML:="roles/download/defaults/main.yml"}
: ${KUBE_VERSION_YAML:="roles/kubespray-defaults/defaults/main.yaml"}

mkdir -p ${TEMP_DIR}
generate_versions() {
    # ARCH used in convert {%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%} to {{arch}}
    if [ "${IMAGE_ARCH}" != "amd64" ]; then ARCH="${IMAGE_ARCH}"; fi

    cat > ${TEMP_DIR}/version.sh << EOF
arch=${ARCH}
image_arch=${IMAGE_ARCH}
ansible_system=${ANSIBLE_SYSTEM}
ansible_architecture=${ANSIBLE_ARCHITECTURE}
EOF
    grep 'kube_version:' ${REPO_ROOT_DIR}/${KUBE_VERSION_YAML} \
    | sed 's/: /=/g' >> ${TEMP_DIR}/version.sh
    grep '_version:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
    | sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/version.sh
    sed -i 's/kube_major_version=.*/kube_major_version=${kube_version%.*}/g' ${TEMP_DIR}/version.sh
    sed -i 's/crictl_version=.*/crictl_version=${kube_version%.*}.0/g' ${TEMP_DIR}/version.sh
}

generate_files_list() {
    echo "source ${TEMP_DIR}/version.sh" > ${TEMP_DIR}/files.sh
    grep 'download_url:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
    | sed 's/: /=/g;s/ //g;s/{{/${/g;s/}}/}/g;s/|lower//g;s/^.*_url=/echo /g' >> ${TEMP_DIR}/files.sh
    bash ${TEMP_DIR}/files.sh | sort > ${TEMP_DIR}/files.list
}

generate_images_list() {
    KUBE_IMAGES="kube-apiserver kube-controller-manager kube-scheduler kube-proxy"
    echo "source ${TEMP_DIR}/version.sh" > ${TEMP_DIR}/images.sh
    grep -E '_repo:|_tag:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
    | sed "s#{%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%}#{{arch}}#g" \
    | sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/images.sh
    sed -n '/^downloads:/,/download_defaults:/p' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
    | sed -n "s/repo: //p;s/tag: //p" | tr -d ' ' | sed 's/{{/${/g;s/}}/}/g' \
    | sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/^/echo /g' >> ${TEMP_DIR}/images.sh
    echo "${KUBE_IMAGES}" | tr ' ' '\n' | xargs -L1 -I {} \
    echo 'echo ${kube_image_repo}/{}:${kube_version}' >> ${TEMP_DIR}/images.sh
    bash ${TEMP_DIR}/images.sh | sort > ${TEMP_DIR}/images.list
}

generate_versions
generate_files_list
generate_images_list
wget -x -P ${TEMP_DIR}/files -i ${TEMP_DIR}/files.list

    The final result of the download is as follows, which basically maintains the original URL path and facilitates subsequent updates and version iterations.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

    temp/files
├── get.helm.sh
│   └── helm-v3.5.4-linux-amd64.tar.gz
├── github.com
│   ├── containerd
│   │   └── nerdctl
│   │       └── releases
│   │           └── download
│   │               └── v0.8.0
│   │                   └── nerdctl-0.8.0-linux-amd64.tar.gz
│   ├── containernetworking
│   │   └── plugins
│   │       └── releases
│   │           └── download
│   │               └── v0.9.1
│   │                   └── cni-plugins-linux-amd64-v0.9.1.tgz
│   ├── containers
│   │   └── crun
│   │       └── releases
│   │           └── download
│   │               └── 0.19
│   │                   └── crun-0.19-linux-amd64
│   ├── coreos
│   │   └── etcd
│   │       └── releases
│   │           └── download
│   │               └── v3.4.13
│   │                   └── etcd-v3.4.13-linux-amd64.tar.gz
│   ├── kata-containers
│   │   └── runtime
│   │       └── releases
│   │           └── download
│   │               └── 1.12.1
│   │                   └── kata-static-1.12.1-x86_64.tar.xz
│   ├── kubernetes-sigs
│   │   └── cri-tools
│   │       └── releases
│   │           └── download
│   │               └── v1.20.0
│   │                   └── crictl-v1.20.0-linux-amd64.tar.gz
│   └── projectcalico
│       ├── calico
│       │   └── archive
│       │       └── v3.17.3.tar.gz
│       └── calicoctl
│           └── releases
│               └── download
│                   └── v3.17.3
│                       └── calicoctl-linux-amd64
└── storage.googleapis.com
    └── kubernetes-release
        └── release
            └── v1.20.6
                └── bin
                    └── linux
                        └── amd64
                            ├── kubeadm
                            ├── kubectl
                            └── kubelet

Get the images you need for deployment

For offline deployment, kubespray support is not very friendly. For example, to get the list of mirrors needed for deployment, the current solution is to deploy a cluster first and then get some resources to get the mirrors used by the pod through kubectl. Personally, I think this approach can be modified, for example, by generating a list of mirrors from the kubespray source code. The following is just a simple list of mirrors, with the following content

  • images.list

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    docker.io/nginx:1.19.0
docker.io/calico/cni:v3.17.3
docker.io/calico/node:v3.17.3
docker.io/calico/kube-controllers:v3.17.3
quay.io/coreos/flannel:v0.13.0
quay.io/coreos/flannel:v0.13.0-amd64
k8s.gcr.io/pause:3.2
k8s.gcr.io/coredns:1.7.0
k8s.gcr.io/kube-apiserver:v1.20.6
k8s.gcr.io/kube-controller-manager:v1.20.6
k8s.gcr.io/kube-proxy:v1.20.6
k8s.gcr.io/kube-scheduler:v1.20.6
k8s.gcr.io/dns/k8s-dns-node-cache:1.17.1
k8s.gcr.io/cpa/cluster-proportional-autoscaler-amd64:1.8.3

    Since the master branch code is always being updated, the current version of the master branch may not be the same as the one here, so you need to change it to the version you need.

  • Based on the list of mirrors above, use skopeo to sync the mirrors to your own mirror repository, such as my hub.k8s.li

    1

    for image in $(cat images.list); do skopeo copy docker://${image} docker://hub.k8s.li/${image#*/}; done

    Synced to my mirror repository, the content will be as follows, by modifying the address of some mirror repositories during the deployment

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    hub.k8s.li/nginx:1.19.0
hub.k8s.li/calico/cni:v3.17.3
hub.k8s.li/calico/node:v3.17.3
hub.k8s.li/calico/kube-controllers:v3.17.3
hub.k8s.li/coreos/flannel:v0.13.0
hub.k8s.li/coreos/flannel:v0.13.0-amd64
hub.k8s.li/pause:3.2
hub.k8s.li/coredns:1.7.0
hub.k8s.li/kube-apiserver:v1.20.6
hub.k8s.li/kube-controller-manager:v1.20.6
hub.k8s.li/kube-proxy:v1.20.6
hub.k8s.li/kube-scheduler:v1.20.6
hub.k8s.li/dns/k8s-dns-node-cache:1.17.1
hub.k8s.li/cpa/cluster-proportional-autoscaler-amd64:1.8.3

The preparations are now largely complete, so let’s start configuring some parameters and inventory files for kubespray

Configuration

Follow the kubespray documentation to make a copy of the inventory/sample directory and then control the deployment by modifying the parameters there.

1

root@debian:/root/kubespray git:(master*) # cp -rf inventory/sample deploy

inventory

  • deploy/inventory

    Create a host inventory file in the following format.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

    [all:vars]
ansible_port=22
ansible_user=root

ansible_ssh_private_key_file=/kubespray/.ssh/id_rsa

[all]
kube-control-1 ansible_host=192.168.4.11
kube-control-2 ansible_host=192.168.4.12
kube-control-3 ansible_host=192.168.4.13
kube-node-1 ansible_host=192.168.4.4

[kube_control_plane]
kube-control-1
kube-control-2
kube-control-3

[etcd]
kube-control-1
kube-control-2
kube-control-3

[kube-node]
kube-control-1
kube-control-2
kube-control-3
kube-node-1

[calico-rr]

[k8s-cluster:children]
kube_control_plane
kube-node
calico-rr

  • ssh mutual trust

    Kubespray uses ansible’s synchronize module to distribute files, based on the rsync protocol, so you must use ssh key pairs to connect to cluster nodes. inventory is configured as a path within the kubespray container, so you need to copy the ssh public and private keys to the repo’s .ssh directory. If the node does not have ssh-free login, you can use the authorized_key module of ansible to add the ssh public key to the authorized_key of the host. The procedure is as follows.

    1
2
3
4
5
6
7

    root@debian:/root/kubespray git:(master*) # mkdir -p .ssh

# 生成 ssh 密钥对
root@debian:/root/kubespray git:(master*) # ssh-keygen -t rsa -f .ssh/id_rsa -P ""

# 将 ssh 公钥添加到所有主机
root@debian:/root/kubespray git:(master*) # ansible -i deploy/inventory all -m authorized_key -a "user={{ ansible_user }} key='{{ lookup('file', '{{ ssh_cert_path }}') }}'" -e ssh_cert_path=./.ssh/id_rsa.pub -e ansible_ssh_pass=passwd

vars

Create and modify the following configuration file

  • deploy/env.yml

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

    ---
# 定义一些组件的版本
kube_version: v1.20.6
calico_version: "v3.17.3"
pod_infra_version: "3.2"
nginx_image_version: "1.19"
coredns_version: "1.7.0"
image_arch: "amd64"

# docker registry domain
registry_domain: "hub.k8s.li"

# file download server url
download_url: "https://dl.k8s.li"

# docker-ce-repo mirrors
docker_mirrors_url: "https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux"

container_manager: "containerd"

# 由于使用的是 containerd 作为 CRI，目前 etcd 不支持 containerd 容器化部署因此需要将该参数修改为 host ，使用 systemd 来部署
etcd_deployment_type: host
etcd_cluster_setup: true
etcd_events_cluster_setup: true
etcd_events_cluster_enabled: true

# kubernetes CNI type 配置集群 CNI 使用的类型
kube_network_plugin: canal

  • deploy/group_vars/all/download.yml

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

    ## Container registry define
gcr_image_repo: "{{ registry_domain }}"
kube_image_repo: "{{ registry_domain }}"
docker_image_repo: "{{ registry_domain }}"
quay_image_repo: "{{ registry_domain }}"

# Download URLs
kubelet_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
kubectl_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
kubeadm_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
etcd_download_url: "{{ download_url }}/github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "{{ download_url }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "{{ download_url }}/github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
calico_crds_download_url: "{{ download_url }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"
crictl_download_url: "{{ download_url }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
helm_download_url: "{{ download_url }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
crun_download_url: "{{ download_url }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
kata_containers_download_url: "{{ download_url }}/github.com/kata-containers/runtime/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"
nerdctl_download_url: "{{ download_url }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

docker-ce mirrors

When kubespray installs docker or containerd containers to run, you need to use the docker-ce source, and you can use Tsinghua’s image source in China. Depending on the Linux distribution, you can add these parameters to the deploy/group_vars/all/offline.yml file. The parameter docker_mirrors_url is the parameter set in env.yml.

  • CentOS/Redhat

    1
2
3
4
5
6
7

    ## CentOS/Redhat
### For EL7, base and extras repo must be available, for EL8, baseos and appstream
### By default we enable those repo automatically
# rhel_enable_repos: false
### Docker / Containerd
docker_rh_repo_base_url: "{{ docker_mirrors_url }}/centos/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable"
docker_rh_repo_gpgkey: "{{ docker_mirrors_url }}/centos/gpg"

  • Fedora

    1
2
3
4
5
6
7

    ## Fedora
### Docker
docker_fedora_repo_base_url: "{{ docker_mirrors_url }}/fedora/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable"
docker_fedora_repo_gpgkey: "{{ docker_mirrors_url }}/fedora/gpg"
### Containerd
containerd_fedora_repo_base_url: "{{ docker_mirrors_url }}/fedora/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable"
containerd_fedora_repo_gpgkey: "{{ docker_mirrors_url }}/fedora/gpg"

  • debian

    1
2
3
4
5
6
7
8

    ## Debian
### Docker
docker_debian_repo_base_url: "{{ docker_mirrors_url }}/debian"
docker_debian_repo_gpgkey: "{{ docker_mirrors_url }}/debian/gpg"
### Containerd
containerd_debian_repo_base_url: "{{ docker_mirrors_url }}/debian"
containerd_debian_repo_gpgkey: "{{ docker_mirrors_url }}/debian/gpg"
# containerd_debian_repo_repokey: 'YOURREPOKEY'

  • ubuntu

    1
2
3
4
5
6
7

    ## Ubuntu
### Docker
docker_ubuntu_repo_base_url: "{{ docker_mirrors_url }}/ubuntu"
docker_ubuntu_repo_gpgkey: "{{ docker_mirrors_url }}/ubuntu/gpg"
### Containerd
containerd_ubuntu_repo_base_url: "{{ docker_mirrors_url }}/ubuntu"
containerd_ubuntu_repo_gpgkey: "{{ docker_mirrors_url }}/ubuntu/gpg"

Deployment

After preparing the configuration above, it’s time to start the deployment. When using ansible for deployment, I prefer to operate in a kubespray container rather than installing python3 on the local development machine. For offline deployments, it is more convenient to build the image in advance and use a docker container.

  • Build Mirror

    1

    root@debian:/root/kubespray git:(master*) # docker build -t kubespray:v2.15.1-kube-v1.20.6 .

  • Run the kubespray container

    1

    root@debian:/root/kubespray git:(master*) # docker run --rm -it --net=host -v $PWD:/kubespray kubespray:v2.15.1-kube-v1.20.6 bash

  • Test if the host is connected properly

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    root@debian:/kubespray# ansible -i cluster/inventory all -m ping
kube-control-3 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
kube-control-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
kube-node-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
kube-control-2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

  • Start cluster deployment

    1

    root@debian:/kubespray# ansible-playbook -i deploy/inventory -e "@deploy/env.yml" cluster.yml

  • The deployment completion log is as follows, when failed is 0, it means the tasks have been successfully run

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    PLAY RECAP ******************************************************************
kube-control-1             : ok=526  changed=67   unreachable=0    failed=0    skipped=978  rescued=0    ignored=0
kube-control-2             : ok=524  changed=66   unreachable=0    failed=0    skipped=980  rescued=0    ignored=0
kube-control-3             : ok=593  changed=76   unreachable=0    failed=0    skipped=1125 rescued=0    ignored=1
kube-node-1                : ok=366  changed=34   unreachable=0    failed=0    skipped=628  rescued=0    ignored=0
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Wednesday 28 April 2021  10:57:57 +0000 (0:00:00.115)       0:15:21.190 *******
===============================================================================
kubernetes/control-plane : kubeadm | Initialize first master -------------- 65.88s
kubernetes/control-plane : Joining control plane node to the cluster. ----- 50.05s
kubernetes/kubeadm : Join to cluster -------------------------------------- 31.54s
download_container | Download image if required --------------------------- 24.38s
reload etcd --------------------------------------------------------------- 20.56s
Gen_certs | Write etcd member and admin certs to other etcd nodes --------- 19.32s
Gen_certs | Write node certs to other etcd nodes -------------------------- 19.14s
Gen_certs | Write etcd member and admin certs to other etcd nodes --------- 17.45s
network_plugin/canal : Canal | Create canal node manifests ---------------- 15.41s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS Template ----- 13.27s
kubernetes/control-plane : Master | wait for kube-scheduler --------------- 11.97s
download_container | Download image if required --------------------------- 11.76s
Gen_certs | Write node certs to other etcd nodes -------------------------- 10.50s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ---------------- 8.28s
policy_controller/calico : Create calico-kube-controllers manifests -------- 7.61s
kubernetes/control-plane : set kubeadm certificate key --------------------- 6.32s
download : extract_file | Unpacking archive -------------------------------- 5.51s
Configure | Check if etcd cluster is healthy ------------------------------- 5.41s
Configure | Check if etcd-events cluster is healthy ------------------------ 5.41s
kubernetes-apps/network_plugin/canal : Canal | Start Resources ------------- 4.85s

  • Cluster Status

    1
2
3
4
5
6

    [root@kube-control-1 ~]# kubectl get node -o wide
NAME             STATUS   ROLES                  AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
kube-control-1   Ready    control-plane,master   5m24s   v1.20.6   192.168.4.11   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.4.4
kube-control-2   Ready    control-plane,master   5m40s   v1.20.6   192.168.4.12   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.4.4
kube-control-3   Ready    control-plane,master   6m28s   v1.20.6   192.168.4.13   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.4.4
kube-node-1      Ready    <none>                 3m53s   v1.20.6   192.168.4.14   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.4.4

  • Cluster Component Status

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

    [root@kube-control-1 ~]# kubectl get all -n kube-system
NAME                                           READY   STATUS             RESTARTS   AGE
pod/calico-kube-controllers-67d6cdb559-cwf62   0/1     CrashLoopBackOff   5          4m10s
pod/canal-node-46x2b                           2/2     Running            0          4m25s
pod/canal-node-5rkhq                           2/2     Running            0          4m25s
pod/canal-node-fcsgn                           2/2     Running            0          4m25s
pod/canal-node-nhkp8                           2/2     Running            0          4m25s
pod/coredns-5d578c6f84-5nnp8                   1/1     Running            0          3m33s
pod/coredns-5d578c6f84-w2kvf                   1/1     Running            0          3m39s
pod/dns-autoscaler-6b675c8995-vp282            1/1     Running            0          3m34s
pod/kube-apiserver-kube-control-1              1/1     Running            0          6m51s
pod/kube-apiserver-kube-control-2              1/1     Running            0          7m7s
pod/kube-apiserver-kube-control-3              1/1     Running            0          7m41s
pod/kube-controller-manager-kube-control-1     1/1     Running            0          6m52s
pod/kube-controller-manager-kube-control-2     1/1     Running            0          7m7s
pod/kube-controller-manager-kube-control-3     1/1     Running            0          7m41s
pod/kube-proxy-5dfx8                           1/1     Running            0          5m17s
pod/kube-proxy-fvrqk                           1/1     Running            0          5m17s
pod/kube-proxy-jd84p                           1/1     Running            0          5m17s
pod/kube-proxy-l2mjk                           1/1     Running            0          5m17s
pod/kube-scheduler-kube-control-1              1/1     Running            0          6m51s
pod/kube-scheduler-kube-control-2              1/1     Running            0          7m7s
pod/kube-scheduler-kube-control-3              1/1     Running            0          7m41s
pod/nginx-proxy-kube-node-1                    1/1     Running            0          5m20s
pod/nodelocaldns-77kq9                         1/1     Running            0          3m32s
pod/nodelocaldns-fn5pd                         1/1     Running            0          3m32s
pod/nodelocaldns-lfjzb                         1/1     Running            0          3m32s
pod/nodelocaldns-xnc6n                         1/1     Running            0          3m32s

NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/coredns   ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP   3m38s

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/canal-node     4         4         4       4            4           <none>                   4m25s
daemonset.apps/kube-proxy     4         4         4       4            4           kubernetes.io/os=linux   7m53s
daemonset.apps/nodelocaldns   4         4         4       4            4           <none>                   3m32s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   0/1     1            0           4m12s
deployment.apps/coredns                   2/2     2            2           3m39s
deployment.apps/dns-autoscaler            1/1     1            1           3m34s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-67d6cdb559   1         1         0       4m12s
replicaset.apps/coredns-5d578c6f84                   2         2         2       3m39s
replicaset.apps/dns-autoscaler-6b675c8995            1         1         1       3m34s