Introduction to Earthly

Earthly is a more advanced Docker image builder, Earthly replaces the traditional Dockerfile with its own Earthfile; Earthfile is as Earthly officially describes:

Makefile + Dockerfile = Earthfile

Earthly supports some Dockerfile extension syntax through buildkit, and integrates Dockerfile with Makefile, making it easier to build and code Dockerfile for multiple platforms; Earthly makes it easier to reuse Dockerfile code and more CI-friendly automatic integration.

Quick Start

2.1. Installing Dependencies

Earthly currently relies on Docker and Git, so make sure you have Docker and Git installed on your machine before installing Earthly.

2.2, Installing Earthly

Earthly is written in Go, so it is mainly a binary file:

1

sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'

After installation Earthly will start a buildkitd container: earthly-buildkitd.

2.3. Syntax highlighting

Earthly officially supports syntax highlighting for VS Code, VIM and Sublime Text, please refer to the official documentation for details.

2.4, basic use

This example is derived from the official Basic tutorial, the following example to compile a Go project as a sample:

first create a directory of any name in which the project source code files and an Earthfile file.

main.go

1
2
3
4
5
6
7

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Earthfile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

FROM golang:1.17-alpine
WORKDIR /go-example

build:
    COPY main.go .
    RUN go build -o build/go-example main.go
    SAVE ARTIFACT build/go-example /go-example AS LOCAL build/go-example

docker:
    COPY +build/go-example .
    ENTRYPOINT ["/go-example/go-example"]
    SAVE IMAGE go-example:latest

Once we have Earthfile we can use Earthly to package it as a mirror.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# 目录结构
~/t/earthlytest ❯❯❯ tree
.
├── Earthfile
└── main.go

0 directories, 2 files

# 通过 earthly 进行构建
~/t/earthlytest ❯❯❯ earthly +docker

Once the build is complete, we can view the image we just built directly from docker’s images list and run:

Advanced Use

3.1. Multi-stage construction

Earthfile contains Makefile-like targets, and the different targets can be referenced by a specific syntax, each target can be executed separately, and earthly will automatically resolve these dependencies during execution.

This multi-stage build time syntax is flexible enough that we can run separate commands at each stage and use different base images; as you can see from the quick start, we always use a base image ( golang:1.17-alpine), and for applications like Go that compile with their own runtime and do not rely on their language SDK, we can in fact put the “distributions” inside a simple runtime image, thus reducing the size of the final image:

Since we use multiple targets, we can run the build target separately to verify our compilation process. This multi-target design allows us to build our application with a finer separation of compilation and packaging steps, and also allows us to verify them separately. For example, we can run the build target separately to verify that our compilation process is correct:

After the other phases are verified, we can run the final target directly, and earthly will automatically recognize the dependency and run its dependent target automatically:

3.2, Extended command

3.2.1, SAVE

The SAVE instruction is one of Earthly’s own extension instructions, actually divided into SAVE ARTIFACT and SAVE IMAGE; where the SAVE ARTIFACT instruction format is as follows:

1

SAVE ARTIFACT [--keep-ts] [--keep-own] [--if-exists] [--force] <src> [<artifact-dest-path>] [AS LOCAL <local-path>]

The SAVE ARTIFACT command is used to save a file or directory from the build runtime environment to the target’s artifact environment; once saved to the artifact environment, it can be referenced in other locations with commands such as COPY, similar to Dockerfile’s COPY --from... syntax; the difference is that SAVE ARTIFACT supports the AS LOCAL <local-path> additional parameter. syntax; the difference is that SAVE ARTIFACT supports the AS LOCAL <local-path> additional parameter, which when specified will make a copy of the file or directory on the host at the same time, generally for debugging purposes. The SAVE ARTIFACT command is shown in the example above, and after running the earthly +build command you will actually see the SAVed ARTIFACT locally:

The other SAVE IMAGE command is used to SAVE the current build environment into an IMAGE, if the -push option is specified and the -push option is added when executing the earthly +target command, the image will be pushed to the target Registry automatically. The -SAVE IMAGE command has the following format:

1

SAVE IMAGE [--cache-from=<cache-image>] [--push] <image-name>...

3.2.2, GIT CLONE

The GIT CLONE command is used to clone a given git repository into the build environment; unlike the RUN git clone... command, GIT CLONE runs through the host git command, it does not depend on the git command inside the container, and you can also configure git authentication directly for earthly to avoid leaking this security information into the build environment; please refer to the official documentation for more information on how to configure git authentication for earthly. here is a sample of the GIT CLONE command:

3.2.3, _COPY

The COPY directive is similar to the standard Dockerfile COPY directive, in addition to supporting the standard Dockerfile COPY function, the COPY directive in **earthly can reference artifacts generated by other target loops, and will automatically declare dependencies when referencing them; that is, when a similar directive exists in the B target If you simply execute earthly +B, then earthly will determine from the dependency analysis that target A needs to be executed before COPY:

1
2
3
4
5

# 与 Dockerfile 相同的使用方式，从上下文复制
COPY [options...] <src>... <dest>

# 扩展支持的从 target 复制方式
COPY [options...] <src-artifact>... <dest>

3.2.4, RUN

The RUN command is consistent with the standard use of Dockerfile, but with more extended options, the command format is as follows:

1
2
3
4
5

# shell 方式运行(/bin/sh -c)
RUN [--push] [--entrypoint] [--privileged] [--secret <env-var>=<secret-ref>] [--ssh] [--mount <mount-spec>] [--] <command>

# exec 方式运行
RUN [[<flags>...], "<executable>", "<arg1>", "<arg2>", ...]

The --privileged option allows running commands to use privileged capabilities, but requires earthly to add the --allow-privileged option when running the target; the --interactive / --interactive- keep option is used to interactively execute some commands, and build continues after the interaction is complete, operations performed during the interaction are persisted to the image:

For reasons of space, please refer to the official documentation Earthfile reference for other specific commands.

3.3、UDCS

UDCs are called “User-defined commands”, i.e. user-defined commands; through UDCs we can strip out specific commands from Earthfile, thus achieving more general and uniform code reuse; here is a sample of defining UDCs commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# 定义一个 Command
# ⚠️ 注意: 语法必须满足以下规则
# 1、名称全大写
# 2、名称下划线分割
# 3、首个命令必须为 COMMAND(后面没有冒号)
MY_COPY:
    COMMAND
    ARG src
    ARG dest=./
    ARG recursive=false
    RUN cp $(if $recursive =  "true"; then printf -- -r; fi) "$src" "$dest"

# target 中引用
build:
    FROM alpine:3.13
    WORKDIR /udc-example
    RUN echo "hello" >./foo
    # 通过 DO 关键字引用 UDCs
    DO +MY_COPY --src=./foo --dest=./bar
    RUN cat ./bar # prints "hello"

Not only can UDCs be defined in an Earthfile, UDCs can be referenced across files and directories:

With UDCs, we can abstract all the operations such as version control of the base image and general handling of special images in this way, and then each Earthfile can refer to them as needed.

3.4, multi-platform build

In the past, when using Dockerfile, we needed to configure and enable buildkit to achieve multi-platform builds; the configuration process may be tedious, but now earthly can help us achieve multi-platform cross-compilation by default, all we need to do is to declare which platforms we need to support in Earthfile:

The above Earthfile will automatically build all four platform images and keep a single tag when running the earthly --push +all build, and will also automatically push to Docker Hub thanks to the -push option:

Summary

Earthly makes up for many of the shortcomings of Dockerfile and solves many of the pain points; however, it may also require some learning costs, but if you are already familiar with Dockerfile, the learning costs are actually not high; so it is still recommended to switch from Dockerfile to Earthfile for unified and versioned management. This article is limited by space (lazy) many places did not speak, such as the shared cache, so more detailed use of Earthly is best to read carefully official documentation.