Offline Deployment
For PaaS toB products, customers often require that the deployment solution must be installed offline, i.e., it cannot rely on any online resources during deployment, such as yum/apt sources for installing some OS packages; container images on docker.io, k8s.gcr.io, quay.io; binary downloads of open source software on GitHub, etc. download files, etc.
As a developer of platform deployment tools, I am always plagued by the challenge of offline deployment. Online container images and binaries are better to solve, because these resources are OS-independent, just download them into the installation package, and start an HTTP server and mirror repository service to provide the download of these resources when deploying.
But for yum/apt and the like, it’s not so simple.
- Firstly, it is not possible to download the packages directly because of their complex dependencies.
- Secondly, even after downloading, it is not possible to install the specified packages directly via yum/apt, although it is possible to copy the packages to the deployment node using scp and install them via rpm or dpkg, but this is not very elegant and the general performance is not very good.
- Finally, there are various Linux distributions and package managers that need to be adapted, and some package names or version numbers vary greatly from one package manager to another, making it impossible to manage them uniformly.
- It is difficult to adapt both arm64 and amd64 sources.
In summary, making offline installers from online yum/apt-type package resources that the platform deployment depends on is a tricky task. I’ve been tossing around this problem for a while and finally found a suitable solution: manage the packages with a YAML configuration file and then use Dockerfile to build them as offline tarballs or container images. If you have similar needs, you can take a look at this solution.
Docker build
The traditional way to make offline sources is to find an appropriate Linux machine, download the packages via the package manager on it, and then create repo index files for those packages.
As you can see this is a very inflexible approach, if I want to create apt offline sources for Debian 9, I need a Debian 9 machine. If I want to adapt multiple Linux distributions, I need multiple OS machines. It’s not easy to manage and use so many different OSes, and container technology, which is now very common, can help us solve this problem. For example, if I want to run a Debian 9 OS, I can just run a container of the Debian 9 image without any additional management costs and with a very light weight.
We often use containers to build backend components written in Golang in our daily work, so can we do the same for building offline sources? We just need to write a Dockerfile for different OS and package managers. Using the docker build multi-stage build feature, you can merge multiple Dockerfiles into one, and then finally copy that build to the same image using COPY -from, such as the nginx container that provides HTTP, or use the BuildKit feature to export as a tarball or as a local directory.
OS Adaptation
Based on my experience in PaaS toB, CentOS is the most popular OS used in the production environment of domestic private cloud customers, followed by Ubuntu and Debian, while RedHat requires a paid subscription, and there are no free images available on DockerHub, so this solution is not guaranteed to work with RedHat. For CentOS, only version 7.9 is required; for Ubuntu, 18.04 and 20.04 are required; for Debian, 9 and 10 are required. 20.04. If you want to support other OS offline sources such as OpenSUSE, you can also refer to this solution to write a Dockerfile file to achieve the adaptation.
Build
The build process is very simple, using a YAML-formatted configuration file to manage the installation of different packages by different package managers or Linux distributions, and doing all the build operations in a single Dockerfile. The source implementation is available at github.com/muzi502/scripts/build-packages-repo.
Build process
Building an offline source using docker build can be roughly divided into the following steps.
- Configuring the yum/apt source inside the build container and installing the tools needed for the build.
- Generate a list of rpm/deb packages on the system and a list of packages that need to be downloaded to solve some package dependency problems.
- downloading the required packages based on the generated package list using the appropriate package manager tool.
- Generate index files for these packages using the appropriate package manager, such as repodata or Packages.gz files.
- COPY the above build products into the same container image, such as nginx; you can also export them as tarballs or directories.
packages.yaml
This file is used to manage the packages that need to be installed by different package managers or Linux distributions. We can divide these packages into 4 categories according to different package managers and distributions.
- common: for packages that have the same name in all package managers or do not require a version, such as vim, curl, wget and other tools. In general, we don’t care about the version of these tools, and the package names of these packages are the same in all package managers, so they can be classified as common packages.
- yum/apt/dnf: This applies to different distributions using the same package manager. For example, if the package for nfs is named nfs-utils in yum but nfs-common in apt, this type of package can be classified as a class.
- OS: for some packages that are unique to that OS, such as installing a package that is available in Ubuntu but not in Debian (e.g. debian-builder or ubuntu-dev-tools).
- OS-distribution codename: The version of such packages is tied to the distribution codename, e.g. `docker-ce=5:19.03.15~3-0~debian-stretch.
|
|
For example, if you want to install the 19.03.15 version of docker-ce in yum, the package name is docker-ce-19.03.15, while in debian the package name is docker-ce=5:19.03.15~3-0 ~debian-stretch . You can use a package manager to see the differences between the same one package such as docker-ce before different package managers, as follows.
|
|
This version number issue is also specially handled in kubespray’s source code, and there is really no good solution to solve it, so we have to maintain this version number manually.
-
roles/container-engine/docker/vars/redhat.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24--- # https://docs.docker.com/engine/installation/linux/centos/#install-from-a-package # https://download.docker.com/linux/centos/<centos_version>>/x86_64/stable/Packages/ # or do 'yum --showduplicates list docker-engine' docker_versioned_pkg: 'latest': docker-ce '18.09': docker-ce-18.09.9-3.el7 '19.03': docker-ce-19.03.15-3.el{{ ansible_distribution_major_version }} '20.10': docker-ce-20.10.5-3.el{{ ansible_distribution_major_version }} 'stable': docker-ce-19.03.15-3.el{{ ansible_distribution_major_version }} 'edge': docker-ce-19.03.15-3.el{{ ansible_distribution_major_version }} docker_cli_versioned_pkg: 'latest': docker-ce-cli '18.09': docker-ce-cli-18.09.9-3.el7 '19.03': docker-ce-cli-19.03.15-3.el{{ ansible_distribution_major_version }} '20.10': docker-ce-cli-20.10.5-3.el{{ ansible_distribution_major_version }} docker_package_info: enablerepo: "docker-ce" pkgs: - "{{ containerd_versioned_pkg[containerd_version | string] }}" - "{{ docker_cli_versioned_pkg[docker_cli_version | string] }}" - "{{ docker_versioned_pkg[docker_version | string] }}" -
roles/container-engine/docker/vars/ubuntu.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20# https://download.docker.com/linux/ubuntu/ docker_versioned_pkg: 'latest': docker-ce '18.09': docker-ce=5:18.09.9~3-0~ubuntu-{{ ansible_distribution_release|lower }} '19.03': docker-ce=5:19.03.15~3-0~ubuntu-{{ ansible_distribution_release|lower }} '20.10': docker-ce=5:20.10.5~3-0~ubuntu-{{ ansible_distribution_release|lower }} 'stable': docker-ce=5:19.03.15~3-0~ubuntu-{{ ansible_distribution_release|lower }} 'edge': docker-ce=5:19.03.15~3-0~ubuntu-{{ ansible_distribution_release|lower }} docker_cli_versioned_pkg: 'latest': docker-ce-cli '18.09': docker-ce-cli=5:18.09.9~3-0~ubuntu-{{ ansible_distribution_release|lower }} '19.03': docker-ce-cli=5:19.03.15~3-0~ubuntu-{{ ansible_distribution_release|lower }} '20.10': docker-ce-cli=5:20.10.5~3-0~ubuntu-{{ ansible_distribution_release|lower }} docker_package_info: pkgs: - "{{ containerd_versioned_pkg[containerd_version | string] }}" - "{{ docker_cli_versioned_pkg[docker_cli_version | string] }}" - "{{ docker_versioned_pkg[docker_version | string] }}"
CentOS7
After introducing the package configuration file above, we will then use Dockerfile to build the offline source of these packages based on this packages.yml configuration file. Here is the Dockerfile for building CentOS 7 offline sources.
|
|
In the last FROM image, scratch is specified here, which is a special image name that represents an empty image layer.
You can also put the build directly into the nginx container, so that running the nginx container directly will serve the yum/apt sources
-
To build as a tarball or local directory, you need to enable the
DOCKER_BUILDKIT=1feature for Docker -
The build log is as follows
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19[+] Building 30.9s (13/13) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 109B 0.0s => [internal] load build definition from Dockerfile.centos 0.0s => => transferring dockerfile: 979B 0.0s => [internal] load metadata for docker.io/library/centos:7.9.2009 2.6s => [centos7 1/7] FROM docker.io/library/centos:7.9.2009@sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e 0.0s => [internal] load build context 0.0s => => transferring context: 818B 0.0s => CACHED [centos7 2/7] RUN yum install -q -y yum-utils createrepo centos-release-gluster epel-release curl && yum-config-manager --add-repo https://download.docker.c 0.0s => [centos7 3/7] WORKDIR /centos/7/os/x86_64 0.0s => [centos7 4/7] RUN curl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.9.3/yq_linux_amd64 && chmod a+x /usr/local/bin/yq && curl 3.2s => [centos7 5/7] COPY packages.yaml packages.yaml 0.1s => [centos7 6/7] RUN yq eval packages.yaml -j | jq -r '.common[],.yum[],.centos[]' | sort -u > packages.list && rpm -qa >> packages.list 1.0s => [centos7 7/7] RUN cat packages.list | xargs yumdownloader --resolve && createrepo -d . 21.6s => [stage-1 1/1] COPY --from=centos7 /centos /centos 0.5s => exporting to client 0.7s => => copying files 301.37MB -
The construction products are as follows
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28root@debian:/build # tree centos centos └── 7 └── os └── x86_64 ├── acl-2.2.51-15.el7.x86_64.rpm ├── ansible-2.9.21-1.el7.noarch.rpm ├── at-3.1.13-24.el7.x86_64.rpm ├── attr-2.4.46-13.el7.x86_64.rpm ├── audit-libs-2.8.5-4.el7.x86_64.rpm ├── audit-libs-python-2.8.5-4.el7.x86_64.rpm ├── avahi-libs-0.6.31-20.el7.x86_64.rpm ├── basesystem-10.0-7.el7.centos.noarch.rpm ├── bash-4.2.46-34.el7.x86_64.rpm …………………………………… ├── redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64.rpm ├── repodata │ ├── 28d2fe2d1dbd9b76d3e5385d42cf628ac9fc34d69e151edfe8d134fe6ac6a6d9-primary.xml.gz │ ├── 5264ca1af13ec7c870f25b2a28edb3c2843556ca201d07ac681eb4af7a28b47c-primary.sqlite.bz2 │ ├── 591d9c2d5be714356e8db39f006d07073f0e1e024a4a811d5960d8e200a874fb-other.xml.gz │ ├── c035d2112d55d23a72b6d006b9e86a2f67db78c0de45345e415884aa0782f40c-other.sqlite.bz2 │ ├── cd756169c3718d77201d08590c0613ebed80053f84a2db7acc719b5b9bca866f-filelists.xml.gz │ ├── ed0c5a36b12cf1d4100f90b4825b93dac832e6e21f83b23ae9d9753842801cee-filelists.sqlite.bz2 │ └── repomd.xml ├── yum-utils-1.1.31-54.el7_8.noarch.rpm └── zlib-1.2.7-19.el7_9.x86_64.rpm 4 directories, 368 files
Debian9
The following is a Debian9 build Dockerfile, the process is similar to CentOS, only the package manager is used in a different way, so I won’t do a detailed source code introduction here.
-
Dockerfile.debian
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30FROM debian:stretch-slim as stretch ARG OS_VERSION=stretch ARG ARCH=amd64 ARG DEP_PACKAGES="apt-transport-https ca-certificates curl gnupg aptitude dpkg-dev" RUN apt update -y -q \ && apt install -y --no-install-recommends $DEP_PACKAGES \ && curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg \ && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian ${OS_VERSION} stable" \ | tee /etc/apt/sources.list.d/docker.list > /dev/null \ && apt update -y -q WORKDIR /debian/${OS_VERSION} RUN curl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.9.3/yq_linux_amd64 \ && chmod a+x /usr/local/bin/yq COPY packages.yaml packages.yaml RUN yq eval '.common[],.apt[],.debian[]' packages.yaml | sort -u > packages.list \ && dpkg --get-selections | grep -v deinstall | cut -f1 >> packages.list RUN chown -R _apt /debian/$OS_VERSION \ && cat packages.list | xargs -L1 -I {} apt-cache depends --recurse --no-recommends --no-suggests \ --no-conflicts --no-breaks --no-replaces --no-enhances {} | grep '^\w' | sort -u | xargs apt-get download RUN cd ../ && dpkg-scanpackages $OS_VERSION | gzip -9c > $OS_VERSION/Packages.gz FROM scratch COPY --from=builder /debian /debian
Ubuntu
The steps to create an Ubuntu offline source are not too different from Debian, just a simple modification of Debian’s Dockerfile should be OK, like 's/debian/ubuntu/g', after all Debian is Ubuntu’s father ~~, so apt uses almost the same way and package names, so I won’t go into it here.
All-in-Oone
By combining the Dockerfile of the above Linux distributions into one, you can build the offline sources for all the OS you need with just one docker build command.
- Dockerfile
|
|
Use
After building the offline sources, run an Nginx service on the deployed machine to provide HTTP downloads of these packages, and configure the machine’s package manager repo configuration file.
-
CentOS 7
-
Debian 9 stretch
1deb [trusted=yes] http://172.20.0.10:8080/debian stretch/ -
Debian 10 buster
1deb [trusted=yes] http://172.20.0.10:8080/debian buster/ -
Ubuntu 18.04 bionic
1deb [trusted=yes] http://172.20.0.10:8080/ubuntu bionic/ -
Ubuntu 20.04 focal
1deb [trusted=yes] http://172.20.0.10:8080/debian focal/
GitHub Action Automated Builds
Once you have the above Dockerfile ready, it’s time to think about the build. For a PaaS or IaaS product, you need to adapt it to mainstream Linux distributions, and sometimes to machines with arm64 architecture. If you build it locally with a manual docker build, it’s not very efficient. So we need to use GitHub actions to automatically build the offline source of these rpm/deb packages, as described in k8sli/os-packages
Code Structure
The build directory holds Dockerfiles for various distributions, and since different distributions and version building methods vary widely for each distribution, each distribution OS is built in a separate Dockerfile.
|
|
Workflow
-
Trigger method
-
Global Variables
-
To build the matrix, each of these jobs will run a runner to do a parallel build
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25jobs: build: runs-on: ubuntu-20.04 strategy: fail-fast: false matrix: include: - name: ubuntu-bionic image_name: os-packages-ubuntu1804 dockerfile: build/Dockerfile.os.ubuntu1804 - name: ubuntu-focal image_name: os-packages-ubuntu2004 dockerfile: build/Dockerfile.os.ubuntu2004 - name: centos-7 image_name: os-packages-centos7 dockerfile: build/Dockerfile.os.centos7 - name: centos-8 image_name: os-packages-centos8 dockerfile: build/Dockerfile.os.centos8 - name: debian-buster image_name: os-packages-debian10 dockerfile: build/Dockerfile.os.debian10 - name: debian-stretch image_name: os-packages-debian9 dockerfile: build/Dockerfile.os.debian9 -
checkout the code and configure the buildx build environment
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19steps: - name: Checkout uses: actions/checkout@v2 with: # fetch all git repo tag for define image tag fetch-depth: 0 - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Log in to GitHub Docker Registry uses: docker/login-action@v1 with: registry: ${{ env.IMAGE_REGISTRY }} username: ${{ env.REGISTRY_USER }} password: ${{ env.REGISTRY_TOKEN }} -
Generate a unique mirror tag by using
git describe --tags -
Build the image and push it to the mirror repository, which will be used later when packaging an all-in-one package
1 2 3 4 5 6 7 8 9- name: Build and push os-package images uses: docker/build-push-action@v2 with: context: . push: ${{ github.event_name != 'pull_request' }} file: ${{ matrix.dockerfile }} platforms: linux/amd64,linux/arm64 tags: | ${{ env.IMAGE_REPO }}/${{ matrix.image_name }}:${{ env.IMAGE_TAG }} -
Generate a new Dockerfile and export the image to the local directory
1 2 3 4 5 6 7 8 9 10 11 12- name: Gen new Dockerfile shell: bash run: | echo -e "FROM scratch\nCOPY --from=${{ env.IMAGE_REPO }}/${{ matrix.image_name }}:${{ env.IMAGE_TAG }} / /" > Dockerfile - name: Build kubeplay image to local uses: docker/build-push-action@v2 with: context: . file: Dockerfile platforms: linux/amd64,linux/arm64 outputs: type=local,dest=./ -
Package and upload the final build to GitHub release
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19- name: Prepare for upload package shell: bash run: | mv linux_amd64/resources resources tar -I pigz -cf resources-${{ matrix.image_name }}-${IMAGE_TAG}-amd64.tar.gz resources --remove-files mv linux_arm64/resources resources tar -I pigz -cf resources-${{ matrix.image_name }}-${IMAGE_TAG}-arm64.tar.gz resources --remove-files sha256sum resources-${{ matrix.image_name }}-${IMAGE_TAG}-{amd64,arm64}.tar.gz > resources-${{ matrix.image_name }}-${IMAGE_TAG}.sha256sum.txt - name: Release and upload packages if: startsWith(github.ref, 'refs/tags/') uses: softprops/action-gh-release@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: files: | resources-${{ matrix.image_name }}-${{ env.IMAGE_TAG }}.sha256sum.txt resources-${{ matrix.image_name }}-${{ env.IMAGE_TAG }}-amd64.tar.gz resources-${{ matrix.image_name }}-${{ env.IMAGE_TAG }}-arm64.tar.gz -
All-in-one merges all built images
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63upload: needs: [build] runs-on: ubuntu-20.04 steps: - name: Checkout uses: actions/checkout@v2 with: # fetch all git repo tag for define image tag fetch-depth: 0 - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Log in to GitHub Docker Registry uses: docker/login-action@v1 with: registry: ${{ env.IMAGE_REGISTRY }} username: ${{ env.REGISTRY_USER }} password: ${{ env.REGISTRY_TOKEN }} - name: Prepare for build images shell: bash run: | git describe --tags --always | sed 's/^/IMAGE_TAG=/' >> $GITHUB_ENV source $GITHUB_ENV echo "FROM scratch" > Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-ubuntu1804:${IMAGE_TAG} / /" >> Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-ubuntu2004:${IMAGE_TAG} / /" >> Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-centos7:${IMAGE_TAG} / /" >> Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-centos8:${IMAGE_TAG} / /" >> Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-debian9:${IMAGE_TAG} / /" >> Dockerfile echo "COPY --from=${{ env.IMAGE_REPO }}/os-packages-debian10:${IMAGE_TAG} / /" >> Dockerfile - name: Build os-packages images to local uses: docker/build-push-action@v2 with: context: . file: Dockerfile platforms: linux/amd64,linux/arm64 outputs: type=local,dest=./ - name: Prepare for upload package shell: bash run: | mv linux_amd64/resources resources tar -I pigz -cf resources-os-packages-all-${IMAGE_TAG}-amd64.tar.gz resources --remove-files mv linux_arm64/resources resources tar -I pigz -cf resources-os-packages-all-${IMAGE_TAG}-arm64.tar.gz resources --remove-files sha256sum resources-os-packages-all-${IMAGE_TAG}-{amd64,arm64}.tar.gz > resources-os-packages-all-${IMAGE_TAG}.sha256sum.txt - name: Release and upload packages if: startsWith(github.ref, 'refs/tags/') uses: softprops/action-gh-release@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: files: | resources-os-packages-all-${{ env.IMAGE_TAG }}.sha256sum.txt resources-os-packages-all-${{ env.IMAGE_TAG }}-amd64.tar.gz resources-os-packages-all-${{ env.IMAGE_TAG }}-arm64.tar.gz
Optimization
Dockerfile
You can consider merging the build process in Dockerfile into a shell script, and then call this script in Dockerfile, which can optimize the maintainability of Dockerfile code and reuse some of the same code when adapting to multiple OSes, but this may lead to invalidation of the docker build cache.
Of course, you can also use a script to merge multiple Dockerfiles into one, as follows.
In fact, if you use GitHub actions to build, you don’t need to merge, you can build in parallel using the actions matrix build feature.
Package version
For some packages that contain Linux distribution designators, it is not convenient to maintain the designators manually, so you can consider changing them to placeholder variables by using sed to replace them after the package.list file is generated in the build container, as follows.
Use sed to process these placeholder variables in the resulting packages.list
|
|
Although this is unsightly, it does work 😂 and eventually we get the correct version number. Anyway, we try to maintain as few package versions as possible, for example, by putting a certain version of the docker-ce package in apt in the configuration file instead of debian/ubuntu, and adding these special items automatically through some environment variables or shell scripts, which can reduce some maintenance costs.
Tips
- When Fedora specifies the package version, you also need to add the version of Fedora
- Some packages in CentOS 7 and CentOS 8 have different package names, so you need to deal with them separately.
- CentOS 7 and CentOS 8 build method is different, the final generation of repodata when CentOS 8 need to deal with a separate
- Fedora 33 and Fedora 34 using GitHub actions build when the arm64 architecture will always be stuck, is due to the buildx bug caused, so only gave the Dockerfile, not put in the GitHub actions build pipeline.