Recently I was rebuilding a very old build process, the program was written by golang, and after building out the binary and running it on the server, there was an exception.

1
2

root@c2a4d003e0d6:/workspace/.build# ./spex_linux_amd64
./spex_linux_amd64: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./spex_linux_amd64)

This is interesting, doesn’t Golang have all its dependencies static linked? How come there is a dynamic link of Glibc dependency?

1
2
3
4
5
6
7
8

root@c2a4d003e0d6:/workspace/.build# ldd spex_linux_amd64
./spex_linux_amd64: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./spex_linux_amd64)
        linux-vdso.so.1 =>  (0x00007ffdd4c82000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8deae5c000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8deab53000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8dea94f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8dea585000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8deb079000)

A more interesting question is why the binary from the old pipeline build does not have these statically linked dependencies?

I downloaded a product from a previous build.

1
2
3
4

root@c2a4d003e0d6:/workspace/.build# file spex
spex: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped
root@c2a4d003e0d6:/workspace/.build# ldd spex
        not a dynamic executable

After some research, I found that the dynamic link is actually from a kafka client confluent-kafka-go, which is based on the c language The client is based on librdkafka, so it should be compiled with CGO_ENABLED=1. Then the dynamic link will come out when it is compiled.

But why is the product of the original pipeline build statically linked?

I recreated the build environment in this pile of bash scripts. Finally, I found that even though it was the exact same environment, the product I built still had dynamic links. The pipeline build does not! This is really amazing.

While doubting my life, I directly changed the CI and added two debug commands after compiling. The miraculous thing happened again, the product of this CI build is also dynamically linked!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

+ ldd .build/spex
    linux-vdso.so.1 =>  (0x00007ffd8d4a8000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f51d8697000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f51d838e000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f51d818a000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f51d7f82000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f51d7bb8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f51d88b4000)
+ objdump -p .build/spex
+ grep NEEDED
  NEEDED               libpthread.so.0
  NEEDED               libm.so.6
  NEEDED               libdl.so.2
  NEEDED               librt.so.1
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2
...

But the binary I downloaded from the file server is obviously not an executable with a dynamic link。

The step after compilation is upload_binary, which I thought should be very simple because its name was only upload. But now I have a vague feeling that it is not simple, and that there is something fishy in this function. So I started reading the script here.

Then I found an amazing command. After the script was compiled, it ssh’d to the file server and executed an upx command.

upx

upx is a tool for compressing binary, as shown above, which reduces the size of all these binary by 46%.

However, the upx compression will make the executable of this dynamic link look like a static link.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

root@c2a4d003e0d6:/workspace/.build# upx spex
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013
 
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  37517592 ->  19717828   52.56%  linux/ElfAMD   spex
 
Packed 1 file.
root@c2a4d003e0d6:/workspace/.build# file spex
spex: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped
root@c2a4d003e0d6:/workspace/.build# ldd spex
        not a dynamic executable

The so-called “seemingly” means that when executing, you actually have to “unpack” and then dynamically link some libs. You can test it by deleting the dependent .so files.

It won’t work anymore.

1
2
3

root@c2a4d003e0d6:/workspace/.build# mv /lib/x86_64-linux-gnu/libm.so.6 /tmp
root@c2a4d003e0d6:/workspace/.build# ./spex --version
./spex: error while loading shared libraries: libm.so.6: cannot open shared object file: No such file or directory

So, in fact, the fundamental reason why the binary built by the new pipeline does not work properly is that

  • The previous binary was also a dynamic link, but after upx it looks like a static link binary now. However, since the previous build environment was very old and relied on a very low version of glibc, it could be run directly on the server.
  • But the new pipeline is built on golang:latest and depends on glibc version 2.29, which is not available on the server, so it won’t work.

After talking to dev, this CGO dependency is necessary. The next solutions are.

  1. use golang’s kafka lib: dev will need to change the code to switch the kafka sdk used. this can be an alternative.
  2. lower the glibc version of golang:latest: distributions generally fix glibc to compile other toolchains, replacing glibc is not wise. Although there are tools like yum downgrade glibc\* that can help with this.
  3. change to an older glibc image: again, you can’t avoid a bunch of old bash scripts.
  4. link c dependencies statically.

To sum up, I still plan to use the latest image to compile, but will depend on all static links, so that once compiled, run everywhere, download it and run.

Static linking of CGO dependencies

If glibc is used, it is not statically linkable.

1
2
3
4
5
6

root@f88271a666f9:/workspace# go build -ldflags "-linkmode external -extldflags '-static'" ./cmd/spex
# git.garena.com/shopee/platform/spex/cmd/spex
/usr/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go@v1.5.2/kafka/librdkafka/librdkafka_glibc_linux.a(rddl.o): in function `rd_dl_open':
(.text+0x1d): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-883441031/000004.o: in function `_cgo_26061493d47f_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Because glibc depends on libnss, which supports a different provider, it must be dynamically linked.

So the only way to replace glibc here is to use musl. librdkafka and the golang package confluent-kafka-go both support musl builds (just specify –tags musl when building) alpine is a distribution based on musl, so here you can build directly with alpine Linux.

Then specify external ld and -static for flags, and the compiled binary will be completely statically linked. The compilation process is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ docker run -it -v $(pwd):/workspace -v /Users/xintao.lai/.netrc:/root/.netrc golang:alpine3.14
/go $ cd /workspace/
/workspace $ apk add git alpine-sdk
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/37) Installing fakeroot (1.25.3-r3)
(2/37) Installing openssl (1.1.1l-r0)
(3/37) Installing libattr (2.5.1-r0)
(4/37) Installing attr (2.5.1-r0)
(5/37) Installing libacl (2.2.53-r0)
(6/37) Installing tar (1.34-r0)
(7/37) Installing pkgconf (1.7.4-r0)
...
/workspace $ go build -ldflags "-linkmode external -extldflags '-static'" -tags musl ./cmd/spex
/workspace $ ldd spex
/lib/ld-musl-x86_64.so.1: spex: Not a valid dynamic program

Static compilation of CGO dependencies can be found in this tutorial: Using CGO bindings under Alpine, CentOS and Ubuntu and this example: go-static- linking.