Recently I was rebuilding a very old build process, the program was written by golang, and after building out the binary and running it on the server, there was an exception.
This is interesting, doesn’t Golang have all its dependencies static linked? How come there is a dynamic link of Glibc dependency?
A more interesting question is why the binary from the old pipeline build does not have these statically linked dependencies?
I downloaded a product from a previous build.
After some research, I found that the dynamic link is actually from a kafka client confluent-kafka-go, which is based on the c language The client is based on librdkafka, so it should be compiled with
CGO_ENABLED=1. Then the dynamic link will come out when it is compiled.
But why is the product of the original pipeline build statically linked?
I recreated the build environment in this pile of bash scripts. Finally, I found that even though it was the exact same environment, the product I built still had dynamic links. The pipeline build does not! This is really amazing.
While doubting my life, I directly changed the CI and added two debug commands after compiling. The miraculous thing happened again, the product of this CI build is also dynamically linked!
But the binary I downloaded from the file server is obviously not an executable with a dynamic link。
The step after compilation is
upload_binary, which I thought should be very simple because its name was only upload. But now I have a vague feeling that it is not simple, and that there is something fishy in this function. So I started reading the script here.
Then I found an amazing command. After the script was compiled, it ssh’d to the file server and executed an
upx is a tool for compressing binary, as shown above, which reduces the size of all these binary by 46%.
However, the upx compression will make the executable of this dynamic link look like a static link.
The so-called “seemingly” means that when executing, you actually have to “unpack” and then dynamically link some libs. You can test it by deleting the dependent
It won’t work anymore.
So, in fact, the fundamental reason why the binary built by the new pipeline does not work properly is that
- The previous binary was also a dynamic link, but after upx it looks like a static link binary now. However, since the previous build environment was very old and relied on a very low version of glibc, it could be run directly on the server.
- But the new pipeline is built on
golang:latestand depends on glibc version 2.29, which is not available on the server, so it won’t work.
After talking to dev, this CGO dependency is necessary. The next solutions are.
- use golang’s kafka lib: dev will need to change the code to switch the kafka sdk used. this can be an alternative.
- lower the glibc version of golang:latest: distributions generally fix glibc to compile other toolchains, replacing glibc is not wise. Although there are tools like
yum downgrade glibc\*that can help with this.
- change to an older glibc image: again, you can’t avoid a bunch of old bash scripts.
- link c dependencies statically.
To sum up, I still plan to use the latest image to compile, but will depend on all static links, so that once compiled, run everywhere, download it and run.
Static linking of CGO dependencies
If glibc is used, it is not statically linkable.
Because glibc depends on libnss, which supports a different provider, it must be dynamically linked.
So the only way to replace glibc here is to use musl. librdkafka and the golang package confluent-kafka-go both support musl builds (just specify –tags musl when building) alpine is a distribution based on musl, so here you can build directly with alpine Linux.
Then specify external ld and
-static for flags, and the compiled binary will be completely statically linked. The compilation process is as follows.
Static compilation of CGO dependencies can be found in this tutorial: Using CGO bindings under Alpine, CentOS and Ubuntu and this example: go-static- linking.