In the previous article on how to use GitLab CI for CI/CD in a Kubernetes cluster, we basically used the Docker On Docker model for building images, because the Kubernetes cluster uses a container runtime like Docker, so we could mount the host docker.sock file from the host to the container to build the image, but recently we changed the container runtime to Containerd after using Kubernetes version 1.22.X, so there is no Docker service available on the node, this time we need to change the mode of building the image, of course, there are many ways to build the image, we Here we still choose to use Docker to build our Docker image, that is, use the Docker IN Docker mode.

Each time we build an image, GitLab Runner will start a Pod with three containers, one of which is the Docker DIND container running the Docker daemon, and the built container will connect to the Docker daemon running on the same Pod. namespace, the Docker CLI that builds the image can connect directly to the Docker daemon via localhost to build. However, one of the biggest problems with this approach is that each build starts a brand new Docker daemon, resulting in no cached Docker layer layer, which can significantly increase our build time.

Instead of running a sidecar container with Docker DIND service for each Pod, let’s run a standalone Docker DIND container with all the Docker CLIs of the build container connected to this one Docker daemon, where we persist the Docker layer layer, which also serves as a cache.

First, we create a PVC to store the persistent data of Docker, and for performance reasons, we use a Local PV here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-volume
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: docker-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /mnt/k8s/docker  # 数据存储的目录
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1  # 运行在node1节点
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: docker-dind
  name: docker-dind-data
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-volume
  resources:
    requests:
      storage: 5Gi

Then deploy a Docker DIND service using Deployment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-dind
  namespace: kube-ops
  labels:
    app: docker-dind
spec:
  selector:
    matchLabels:
      app: docker-dind
  template:
    metadata:
      labels:
        app: docker-dind
    spec:
      containers:
        - image: docker:dind
          name: docker-dind
          args:
          - --registry-mirror=https://ot2k4d59.mirror.aliyuncs.com/  # 指定一个镜像加速器地址
          env:
            - name: DOCKER_DRIVER
              value: overlay2
            - name: DOCKER_HOST
              value: tcp://0.0.0.0:2375
            - name: DOCKER_TLS_CERTDIR   # 禁用 TLS 
              value: ""
          volumeMounts:
            - name: docker-dind-data-vol # 持久化docker根目录
              mountPath: /var/lib/docker/
          ports:
            - name: daemon-port
              containerPort: 2375
          securityContext:
            privileged: true # 需要设置成特权模式
      volumes:
        - name: docker-dind-data-vol
          persistentVolumeClaim:
            claimName: docker-dind-data

Then create a Service to facilitate the connection of the built Docker CLI to it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: Service
metadata:
  name: docker-dind
  namespace: kube-ops
  labels:
    app: docker-dind
spec:
  ports:
    - port: 2375
      targetPort: 2375
  selector:
    app: docker-dind

After deploying the Docker DIND service, we can use this daemon to build images in Gitlab CI, as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

tages:
  - image

build_image:
  stage: image
  image: docker:latest
  variables:
    DOCKER_HOST: tcp://docker-dind:2375  # 通过 service dns 形式连接 docker dind 服务
  script:
    - docker info
    - docker build -t xxxx .
    - docker push xxxx
  only:
    - tags

Since we cache the Docker layer, the build speed will be significantly faster at this time. Finally, as a lot of images are built, we can write a Cronjob to clear the cache regularly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: batch/v1
kind: CronJob
metadata:
  name: docker-dind-clear-cache
  namespace: kube-ops
spec:
  schedule: 0 0 * * 0  # 每周清理一次
  jobTemplate:
    metadata:
      labels:
        app: docker-dind
      name: docker-dind-clear-cache
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: clear-cache
              image: docker:latest
              command:
                - docker
                - system
                - prune
                - -af
              env:
                - name: DOCKER_HOST
                  value: tcp://docker-dind:2375