1. Objectives

The core goal of this article:

  • Use the full docker cli command on a Mac, including support for basic -v mounts.
  • Support for x86 emulation, with the ability to build or run images for x86.
  • CPU architecture switching where possible, preferably for both arm64 and x86.

2. the choice of tools

First of all, we are most familiar with Docker Desktop, the installation package is huge and the UI is very laggy. Not to mention the startup speed and the occasional jam, so Docker Desktop is not considered at all; so the remaining options are as follows.

  • VM solution
  • Colima solution
  • Lima solution

Let’s start with the conclusion: Lima YES! VM solution is expensive and unpleasant, Colima is temporary and unstable. See Section 5 directly for the Lima solution.

3. Virtual Machine Solution

Currently, the only available or usable virtual machine on M1 is Parallels Desktop, while others like VBox and VMware are still immature; if pure qemu is a bit too hardcore (if you want to package your own scripts, forget it); for Parallels Desktop, we need to buy the development version of the license , because we need to use prlctl to achieve some automation, several hundred a year… After testing this solution is also somewhat feasible:

  1. first create a virtual machine like Ubuntu through PD
  2. install Docker in the virtual machine
  3. start the virtual machine through cli program, and mount ~ rw to the virtual machine

Based on this solution I personally tried, I wrote a small tool PD to help with the mounting action. tool to assist in the mounting process. However, this tool has some obvious drawbacks:

  • x86 emulation is not currently supported, which can be mitigated by binfmt, but is not perfect
  • Virtual machines cost money and require virtual machine cli support

4. Colima solution

Colima is claimed to be a solution for Mac platform containerized toolchain, but actual testing found that Colima is not stable, sometimes there may be some small problems; of course, the biggest problem of Colima is: customizable degree is not high, the underlying layer is based on Lima. Colima specific use and so on here is not described in detail, is not stable Not really recommended.

5. Lima Solution

Lima is currently an automated VM solution based on QEMU, and thanks to its excellent design, Cloud Init can help us to complete the hook at many stages; so it is easy to install Docker or k8s, or get something else; and many solutions such as docker have official samples, so we can copy them directly and use them.

5.1. Lima installation

Lima is relatively easy to install on Mac, the following command will install the master branch version.

1

brew install lima --HEAD

Under normal circumstances, the installation of Lima comes with the installation of QEMU, if QEMU is already installed locally, you may need to execute the following command Upgrade QEMU to 7.0 :

1

brew upgrade qemu

In order to use docker, you also need to install the docker cli via brew:

1

brew install docker

5.2. Using Lima

By default, Lima generates a lima shortcut command after installation, but it is not recommended to use it, because it looks convenient but there is no control over too many parameters, so it is still recommended to use the standard limactl command to operate. limactl is used in the following way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

Lima: Linux virtual machines

Usage:
  limactl [command]

Examples:
  Start the default instance:
  $ limactl start

  Open a shell:
  $ lima

  Run a container:
  $ lima nerdctl run -d --name nginx -p 8080:80 nginx:alpine

  Stop the default instance:
  $ limactl stop

  See also example YAMLs: /opt/homebrew/share/doc/lima/examples

Available Commands:
  completion    Generate the autocompletion script for the specified shell
  copy          Copy files between host and guest
  delete        Delete an instance of Lima.
  edit          Edit an instance of Lima
  factory-reset Factory reset an instance of Lima
  help          Help about any command
  info          Show diagnostic information
  list          List instances of Lima.
  prune         Prune garbage objects
  shell         Execute shell in Lima
  show-ssh      Show the ssh command line
  start         Start an instance of Lima
  stop          Stop an instance
  sudoers       Generate /etc/sudoers.d/lima file for enabling vmnet.framework support
  validate      Validate YAML files

Flags:
      --debug     debug mode
  -h, --help      help for limactl
  -v, --version   version for limactl

Use "limactl [command] --help" for more information about a command.

5.3. Lima configuration file

Lima determines how to create a virtual machine by reading a yaml configuration description file, which has the following basic structure:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101

# 定义每个平台架构需要使用的启动镜像
images:
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"

# 定义虚拟机需要使用哪个架构启动(对应上面的镜像)
arch: "x86_64"

# CPU 数量
cpus: 4

# 内存大小
memory: "16G"

# 磁盘大小
disk: "100G"

# 虚拟机与 macOS 宿主机挂载时使用的挂载技术
# 目前推荐 9p, 可换成 sshfs, 但是 sshfs 会有权限问题
mountType: 9p

# 定义虚拟机和 macOS 宿主机有哪些目录可以共享
mounts:
- location: "~"
  # 定义虚拟机对这个目录是否可写
  writable: true
  9p:
    # 对于可写的共享目录, cache 推荐类型为 mmap, 不写好像默认 fscache
    cache: "mmap"
- location: "/tmp/lima"
  writable: true
  9p:
    cache: "mmap"
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false

# cloud-init hook
provision:
# Define what permissions to execute scripts within the virtual machine
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    if command -v docker >/dev/null 2>&1; then
      docker run --platform=linux/amd64 --privileged --rm tonistiigi/binfmt --install all
      exit 0
    else
      export DEBIAN_FRONTEND=noninteractive
      curl -fsSL https://get.docker.com | sh
      docker run --platform=linux/amd64 --privileged --rm tonistiigi/binfmt --install all
      # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
      systemctl disable --now docker
      apt-get install -y uidmap dbus-user-session
    fi
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
# Self-defined post-launch message output
message: |
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create amd64 --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use amd64
  ------

5.4. Starting a VM

The limactl command provides a start subcommand to start a VM, which takes a single argument, the form of which varies depending on the behavior:

  • If the argument is a file path, the file is assumed to be the yaml configuration of a lima virtual machine, which is read and started.
  • If the argument is a simple string, it first tries to find a virtual machine with the same name among the existing ones, and starts it immediately if it finds one.
  • If the argument is a simple string and no existing VM with the same name is found, then try to create a new VM using the built-in templates.

Using the docker configuration file I defined above as an example, we can create a docker virtual machine by starting this configuration directly:

1

limactl start ./docker-amd64.yaml

After startup, you will be prompted whether to edit and then start again. This is to use the same configuration to start multiple vm’s, so just start them without editing.

limactl

After a few moments the virtual machine will start successfully:

virtual machine start successfully

After booting, execute the two commands printed at the bottom to use docker on the host machine. This essentially uses the docker context feature, and then mounts the sock file from the virtual machine to the host, and configures the docker context for seamless use of docker commands.

5.5. Virtual machine tuning

In some cases, we need to customize the configuration of the VM, mainly by adjusting the provision section of the configuration file; in this section, if mode is defined as system, the command will be executed as root user, otherwise it will be executed as normal user. It is important to note that the scripts we define need to be idempotent, because they will be executed once each time, so generally we need to write the logic for the commands that may cause data erasure actions, to avoid repeated execution.

For file mounting, we recommend using 9p type, future lima will switch to this mount mode completely; also after testing currently only 9p mount mode, local directory rw will not have permission problem when mapping to virtual machine, sshfs mount mode will cause permission error if encountering chown or other commands, which may lead to container boot failure (e.g. mysql).

During test VM configuration, you can use limactl delete -f xxxx to force delete the target VM, and then restart it; The VM name is the same as the yaml file name by default, you can use limactl ls command to check it.

5.6. Multi-platform compatibility

In my docker configuration example above, binfmt is automatically installed every time the virtual machine finishes booting:

1

docker run --platform=linux/amd64 --privileged --rm tonistiigi/binfmt --install all

This ensures that docker images from other platforms can be run regardless of the original architecture of the Lima VM; typically, some openjdk8 images are only available in amd64, but can still be used if the lima VM is aarch64.

In addition to this “faster” cross-architecture approach, lima also supports defining the architecture directly in the VM, so that the target architecture is emulated directly from the VM system level when qemu is booted; this approach has the advantage of being very compatible with the target architecture, but it will run more slowly. Adjusting the VM architecture is a simple matter of modifying the arch configuration (note that the image of the target architecture must be configured):

1
2
3
4
5
6
7
8
9

# Define the boot images that need to be used for each platform architecture
images:
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"

# Define which architecture this VM needs to be started with (corresponding to the image of the target architecture above will be used)
arch: "aarch64"

6. Summary

Overall, Docker Desktop is basically hard to use on mac, Colima is not too mature yet, suitable for light docker users; but heavy docker users with customization needs still recommend Lima virtual machine; Lima also supports many operating systems, there are a lot of official sample templates (including k8s, k3s, podman, etc.), which is very suitable for heavy container users.