Recently I was rebuilding a very old build process, the program was written by golang, and after building out the binary and running it on the server, there was an exception.

 1 2  root@c2a4d003e0d6:/workspace/.build# ./spex_linux_amd64 ./spex_linux_amd64: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by ./spex_linux_amd64) 

This is interesting, doesn’t Golang have all its dependencies static linked? How come there is a dynamic link of Glibc dependency?

 1 2 3 4 5 6 7 8  root@c2a4d003e0d6:/workspace/.build# ldd spex_linux_amd64 ./spex_linux_amd64: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by ./spex_linux_amd64) linux-vdso.so.1 => (0x00007ffdd4c82000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8deae5c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8deab53000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8dea94f000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8dea585000) /lib64/ld-linux-x86-64.so.2 (0x00007f8deb079000) 

A more interesting question is why the binary from the old pipeline build does not have these statically linked dependencies?

I downloaded a product from a previous build.

 1 2 3 4  root@c2a4d003e0d6:/workspace/.build# file spex spex: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped root@c2a4d003e0d6:/workspace/.build# ldd spex not a dynamic executable 

After some research, I found that the dynamic link is actually from a kafka client confluent-kafka-go, which is based on the c language The client is based on librdkafka, so it should be compiled with CGO_ENABLED=1. Then the dynamic link will come out when it is compiled.

But why is the product of the original pipeline build statically linked?

I recreated the build environment in this pile of bash scripts. Finally, I found that even though it was the exact same environment, the product I built still had dynamic links. The pipeline build does not! This is really amazing.

While doubting my life, I directly changed the CI and added two debug commands after compiling. The miraculous thing happened again, the product of this CI build is also dynamically linked!

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17  + ldd .build/spex linux-vdso.so.1 => (0x00007ffd8d4a8000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f51d8697000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f51d838e000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f51d818a000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f51d7f82000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f51d7bb8000) /lib64/ld-linux-x86-64.so.2 (0x00007f51d88b4000) + objdump -p .build/spex + grep NEEDED NEEDED libpthread.so.0 NEEDED libm.so.6 NEEDED libdl.so.2 NEEDED librt.so.1 NEEDED libc.so.6 NEEDED ld-linux-x86-64.so.2 ... 

But the binary I downloaded from the file server is obviously not an executable with a dynamic link。

The step after compilation is upload_binary, which I thought should be very simple because its name was only upload. But now I have a vague feeling that it is not simple, and that there is something fishy in this function. So I started reading the script here.

Then I found an amazing command. After the script was compiled, it ssh’d to the file server and executed an upx command.

upx is a tool for compressing binary, as shown above, which reduces the size of all these binary by 46%.

However, the upx compression will make the executable of this dynamic link look like a static link.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14  root@c2a4d003e0d6:/workspace/.build# upx spex Ultimate Packer for eXecutables Copyright (C) 1996 - 2013 UPX 3.91 Markus Oberhumer, Laszlo Molnar & John Reiser Sep 30th 2013 File size Ratio Format Name -------------------- ------ ----------- ----------- 37517592 -> 19717828 52.56% linux/ElfAMD spex Packed 1 file. root@c2a4d003e0d6:/workspace/.build# file spex spex: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped root@c2a4d003e0d6:/workspace/.build# ldd spex not a dynamic executable 

The so-called “seemingly” means that when executing, you actually have to “unpack” and then dynamically link some libs. You can test it by deleting the dependent .so files.

It won’t work anymore.

 1 2 3  root@c2a4d003e0d6:/workspace/.build# mv /lib/x86_64-linux-gnu/libm.so.6 /tmp root@c2a4d003e0d6:/workspace/.build# ./spex --version ./spex: error while loading shared libraries: libm.so.6: cannot open shared object file: No such file or directory 

So, in fact, the fundamental reason why the binary built by the new pipeline does not work properly is that

• The previous binary was also a dynamic link, but after upx it looks like a static link binary now. However, since the previous build environment was very old and relied on a very low version of glibc, it could be run directly on the server.
• But the new pipeline is built on golang:latest and depends on glibc version 2.29, which is not available on the server, so it won’t work.

After talking to dev, this CGO dependency is necessary. The next solutions are.

1. use golang’s kafka lib: dev will need to change the code to switch the kafka sdk used. this can be an alternative.
2. lower the glibc version of golang:latest: distributions generally fix glibc to compile other toolchains, replacing glibc is not wise. Although there are tools like yum downgrade glibc\* that can help with this.
3. change to an older glibc image: again, you can’t avoid a bunch of old bash scripts.
4. link c dependencies statically.

To sum up, I still plan to use the latest image to compile, but will depend on all static links, so that once compiled, run everywhere, download it and run.

## Static linking of CGO dependencies

If glibc is used, it is not statically linkable.

 1 2 3 4 5 6  root@f88271a666f9:/workspace# go build -ldflags "-linkmode external -extldflags '-static'" ./cmd/spex # git.garena.com/shopee/platform/spex/cmd/spex /usr/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go@v1.5.2/kafka/librdkafka/librdkafka_glibc_linux.a(rddl.o): in function rd_dl_open': (.text+0x1d): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/bin/ld: /tmp/go-link-883441031/000004.o: in function _cgo_26061493d47f_C2func_getaddrinfo': /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking 

Because glibc depends on libnss, which supports a different provider, it must be dynamically linked.

So the only way to replace glibc here is to use musl. librdkafka and the golang package confluent-kafka-go both support musl builds (just specify –tags musl when building) alpine is a distribution based on musl, so here you can build directly with alpine Linux.

Then specify external ld and -static for flags, and the compiled binary will be completely statically linked. The compilation process is as follows.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16  $docker run -it -v$(pwd):/workspace -v /Users/xintao.lai/.netrc:/root/.netrc golang:alpine3.14 /go $cd /workspace/ /workspace$ apk add git alpine-sdk fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz (1/37) Installing fakeroot (1.25.3-r3) (2/37) Installing openssl (1.1.1l-r0) (3/37) Installing libattr (2.5.1-r0) (4/37) Installing attr (2.5.1-r0) (5/37) Installing libacl (2.2.53-r0) (6/37) Installing tar (1.34-r0) (7/37) Installing pkgconf (1.7.4-r0) ... /workspace $go build -ldflags "-linkmode external -extldflags '-static'" -tags musl ./cmd/spex /workspace$ ldd spex /lib/ld-musl-x86_64.so.1: spex: Not a valid dynamic program 

Static compilation of CGO dependencies can be found in this tutorial: Using CGO bindings under Alpine, CentOS and Ubuntu and this example: go-static- linking.