Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.
Distributing images for multiple deployments in a single build dramatically improves the efficiency of application delivery, and for scenarios where applications need to be deployed across platforms, building images across platforms with docker buildx is a fast and efficient solution.
Most mirror hosting platforms support multi-platform mirrors, which means that a single tag in a mirror repository can contain multiple mirrors from different platforms. Take the
python image repository of
docker hub for example, the
3.9.6 tag contains 10 images for different systems and architectures (platform = system + architecture).
When pulling an image with cross-platform support via
docker pull or
docker will automatically select the image that matches the current running platform. Thanks to this feature, when distributing mirrors cross-platform, we don’t need to do anything about the consumption of the mirrors, we only need to care about the production of the mirrors, i.e. how to build cross-platform mirrors.
docker build command cannot do cross-platform build tasks, we need to extend its functionality by installing the
buildx plugin for the
docker command line.
buildx can use the additional features of build images provided by Moby BuildKit to create multiple builder instances, perform build tasks in parallel on multiple nodes, and build across platforms.
Docker Desktop for macOS or Windows, and
docker for Linux distributions installed via the
rpm packages have
buildx built in and do not need to be installed separately.
docker does not have the
buildx command, you can download the binary package to install it.
- First, find the binary for your platform from the release page of the Docker buildx project.
- Download the binaries locally and rename them to
docker-buildxand move them to docker’s plugins directory
- Grant executable permissions to the binaries.
If the local
docker version is higher than 19.03, it is easier to build and install it directly locally with the following command.
buildx is done as follows.
docker build commands are basically the same experience, and also support
build common options such as
docker buildx manages the build configuration and nodes through the builder instance object. The command line sends the build task to the builder instance, which in turn assigns it to a qualified node for execution. We can create multiple builder instances based on the same
docker service application and make them available to different projects to isolate the configuration of each project, or create a builder instance for a set of remote
docker nodes to form a build array and quickly switch between arrays.
A builder instance can be created using the
docker buildx create command, which will create a new builder instance for the node with the currently used docker service. To use a remote node, you can specify the remote port with the
DOCKER_HOST environment variable when creating the example or switch to the remote node’s
docker context in advance. The following creates a new builder instance with a remote node and specifies its driver, target platform, and instance name via command line options.
docker buildx ls will list all available builder instances and the nodes in the instance.
After an instance is created, you can continue to add new nodes to it by using the
-append <node> option of the
docker buildx create command to add nodes to the builder instance specified by the
-name <builder> option.
docker buildx inspect,
docker buildx stop and
docker buildx rm commands are used to manage the lifecycle of an instance.
docker buildx use <builder> will switch to the specified builder instance.
The buildx instance performs build tasks in two ways, the two execution methods are referred to as using different
dockerdriver: executes builds using the BuildKit library integrated in the Docker service application.
docker-containerdriver: starts a container containing BuildKit and executes the build in the container.
docker driver cannot use a small number of
buildx features (such as building multiple platform images at the same time in a single run), and there is also a difference in the default output format of the images: the
docker driver outputs the build results directly to the
docker image directory (usually
/var/lib/ overlay2), and the output images can be listed by executing the
docker images command, while
docker container requires the
--output option to specify the output format as a mirror or other format.
In order to build images for multiple platforms at once, we will use the
docker container-driven builder instance below.
Cross-platform build strategies for ## buildx
Depending on the build node and target language,
buildx supports the following three cross-platform build strategies.
- create a lightweight virtual machine via QEMU’s user-state mode and build the image in the virtual machine system.
- add multiple nodes of different target platforms to a builder instance and build the corresponding platform image from the native nodes.
- build and cross-compile to different target architectures in phases.
While QEMU is typically used to emulate a full operating system, it can also run in user state mode: register a binary conversion handler with
binfmt_misc on the host system and dynamically translate the binary file at runtime, converting system calls from the target CPU architecture to the current system’s CPU architecture as needed. The end result is like running the target CPU architecture’s binaries in a virtual machine. qemu support is built into Docker Desktop, and other platforms that meet the requirements to run it can be installed in the following way.
This approach does not require any changes to the existing Dockerfile and is cheap to implement, but obviously not very efficient.
Adding native nodes from different system architectures to the builder instance can bring better support for cross-platform compilation and is more efficient, but requires sufficient infrastructure support.
If the build project uses a programming language that supports cross-compilation (e.g. C and Go), you can take advantage of the staged build feature provided by Dockerfile: first compile the binaries of the target architecture in the same architecture as the build node, and then copy these binaries to another image of the target architecture. A concrete example is implemented below using Go. This approach does not require additional hardware and yields better performance, but is only possible with specific programming languages.
Build multiple architectures at once Go image practice
Source code and Dockerfile
The following will be a simple Go project as an example, assuming that the sample program file
main.go reads as follows.
The Dockerfile that defines the build process is as follows.
The build process is divided into two phases.
- In one phase, we pull a
golangimage for the same platform as the current build node and compile it into a binary for the target architecture using Go’s cross-compilation features.
- Then pull the
alpineimage for the target platform and copy the compilation results from the previous phase into the image.
Executing a cross-platform build
When executing the build command, in addition to specifying the image name, the other two important options are to specify the target platform and the output format.
-docker buildx build specifies the target platform for the build via the
-platform option, and the FROM command in Dockerfile without the
-platform flag will pull the base image with the target platform, and the resulting image will belong to the target platform. In addition, the value of this option can be used in Dockerfile with the
TARGETARCH parameters. When using the
docker-container driver, this option accepts multiple comma-separated values as input to specify multiple target platforms at the same time, and the build results for all platforms are combined into a single list of images as output, so they cannot be directly output as local
docker images images.
docker buildx build supports rich output behavior. The
--output=[PATH,-,type=TYPE[,KEY=VALUE] option allows you to specify the output type and path of the build result.
- local: the build result will be written to the local path specified by
destin file system format, such as
--output type=local,dest=. /output.
- tar: the build result will be written to the local path specified by
- oci: The build result is written to the local path specified by
destin OCI standard image format.
- docker: The build result is written to the local path specified by
destin Docker standard image format or loaded into
docker’s image repository. This option is not available when multiple target platforms are specified at the same time.
- image: Export as an image or list of images, and support the
push=trueoption to push directly to a remote repository, which can be used when specifying multiple target platforms.
- registry: a compact representation of
For this example, we execute the following
docker buildx build command.
This command will build
linux/arm images in the current directory and push the output directly to the remote Aliyun image repository.
The build process can be broken down as follows.
dockertransfers the build context to the builder instance. 2.
- builder builds images for each target platform specified by the
-platformoption on the command line, including pulling the base image and performing the build steps. 3.
- export the build results, and the image file layer is pushed to the remote repository. 4.
- Generate a manifest JSON file and push it to the remote repository as a mirror tag.
Validate the build results
After running you can probe the images that have been pushed to the remote repository with
docker buildx imagetools at the end of the run.
Finally, pull and run images on different platforms with the
latest tag to verify that the build is correct. When using Docker Desktop, its own integrated virtualization feature allows you to run images from different platforms and pull images directly with
How to cross-compile Golang’s CGO projects
Support for cross-compiling to common operating systems and CPU architectures is one of the great advantages of Golang, but the solution in the above example only works with pure Go code, and the situation becomes more complicated if the project calls C code via
Preparing the cross-compilation environment and dependencies
In order to compile C code to the target platform smoothly, you need to install the C cross-compiler (usually based on
gcc) for the target platform in the compilation environment.
Next, you need to install the C standard libraries for the target platform (usually the standard libraries are installed as dependencies of the cross-compiler and do not need to be installed separately), and depending on the dependencies of the C code you are calling, you may need to install some additional C dependencies (e.g.
libopus-dev or something like that).
We will use the official
golang:1.14 image for the
amd64 architecture as the base image for compilation, and the Linux distribution is Debian. assuming the target platform for cross-compilation is
linux/arm64, the cross-compiler to be prepared is
gcc-aarch64-linux-gnu and the C standard library is
libc6-dev-arm64-cross, and the installer is as follows.
libc6-dev-arm64-cross will be installed at the same time.
Thanks to the multi-architecture installation capabilities provided by the Debian package manager
dpkg, if our code relies on non-standard libraries such as
libopus-dev, we can install their
arm64 architecture installers via
Cross-compiling CGO examples
Suppose the following sample code for
cgo is available.
The Dockerfile that will be used is as follows.
gcc-aarch64-linux-gnu as a cross-compiler via
apt-get, and the example program is simple so no additional dependencies are needed. When running
go build for compilation, you need to specify the cross-compiler to use via the
CC_FOR_TARGET environment variables.
In order to perform multiple builds on the same Dockerfile (assuming the target architecture is only
RUN command at the bottom uses a trick to execute different build commands via Bash’s conditional syntax.
- If the target platform of the build task is
CC_FOR_TARGETenvironment variables as installed cross-compilers (note that their values are different).
- If the target platform for the build task is
amd64, no cross-compiler-related variables are specified, and the default
gccis used as the compiler.
The final command to execute the build using buildx is as follows.
With the help of the
Buildx plugin, we can use
docker to easily build cross-platform application images in the absence of infrastructure.
However, the default way of virtualizing target platform instructions via QEMU has a significant performance bottleneck, and if the language used to write the application supports cross-compilation, we can achieve higher efficiency by combining
buildx with cross-compilation.
This article concludes with a solution for an advanced scenario: how to cross-compile a Golang project that uses CGO, and gives an example of compiling to the