I ran into a problem a while ago, the program somehow crashed and the stack didn’t show anything. Today, I changed a parameter to make the golang program core dump when it crashes.

In fact, the core is to add an environment variable, GOTRACEBACK=1. But there are some other system-related issues, which I will briefly document in this article.

After Golang 1.6, there have been some changes to the optional values of this environment variable, the new values are as follows:

  • GOTRACEBACK=none will suppress all tracebacks, you only get the panic message.
  • GOTRACEBACK=single is the new default behaviour that prints only the goroutine believed to have caused the panic.
  • GOTRACEBACK=all causes stack traces for all goroutines to be shown, but stack frames related to the runtime are suppressed.
  • GOTRACEBACK=system is the same as the previous value, but frames related to the runtime are also shown, this will reveal goroutines started by the runtime itself.
  • GOTRACEBACK=crash is unchanged from Go 1.5.

Some points to note:

First, an introduction to some other very useful environment variables that can control the golang runtime in addition to this GOTRACEBACK parameter, as summarized nicely in this article.

Then this parameter is not valid on macOS, so don’t waste your time on MAC.

Linux is also limited by ulimit. You can check the size limit on Core dumps with ulimit -c. If it’s 0 it won’t dump, and it’s not recommended to set it to ulimited. If the application is started with systemd, you can set the LimitCORE= parameter in the service unit file to have the same effect.

Where is the generated core dump stored?

You can check this with the following command.

1

cat /proc/sys/kernel/core_pattern

This defines how the core dump file is named.

However, in ubuntu, you will see the following output.

1

/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

This means that it is directed to apport via pipe. apport is the core dump management service that the ubuntu distribution has chosen to use.

By default, user programs do not have core dumps. We then have two solutions:

  1. disable apport and use the system’s core dump to write directly to disk
  2. configure apport to also write the user’s core dump file

The first method is relatively simple. Just modify the /proc/sys/kernel/core_pattern file in the above section.

1
2

echo "kernel.core_pattern=/tmp/%e.%t.%p.%s.core" > /proc/sys/kernel/core_pattern
sysctl --system

Note that there is a small issue to be aware of here: this configuration is global and can only be edited by the root account. This will not work under a normal user, because in this line echo is executed with sudo, but the redirection is actually done by the shell (bash), and the redirection, i.e. the actual writing, is not actually done under root, so you will get the error: Permission denied, or Destination /proc /sys/kernel not writable.

1

sudo echo "kernel.core_pattern=/tmp/%e.%t.%p.%s.core" > /proc/sys/kernel/core_pattern

The solution is to use the following command.

1

sudo bash -c 'echo "kernel.core_pattern=/tmp/%e.%t.%p.%s.core" > /proc/sys/kernel/core_pattern'

Then you can disable apport.

1
2

systemctl stop apport
systemctl disable apport

For the second method, first make sure that apport is running. You can check this with systemctl status apport. You can also look at the apport logs.

1

tail -f /var/log/apport.log

Trigger a core dump and you will see:

1

 executable does not belong to a package, ignoring

This means that the core dump is not from an ubuntu package and is ignored.

To configure this, modify the ~/.config/apport/settings (if you don’t have it, create it manually) file and write the following.

1
2

[main]
unpackaged=true

Trigger another core dump, and this time the log will have the information written to it.

1
2
3

ERROR: apport (pid 3709) Thu Jun 22 07:00:58 2023: called for pid 3704, signal 6, core limit 0, dump mode 1
ERROR: apport (pid 3709) Thu Jun 22 07:00:58 2023: executable: /tmp/go-build2218735647/b001/exe/crash (command line "/tmp/go-build2218735647/b001/exe/crash")
ERROR: apport (pid 3709) Thu Jun 22 07:00:59 2023: wrote report /var/crash/_tmp_go-build2218735647_b001_exe_crash.0.crash

Note also that this file is not a core dump file, but a debug file packed by apport and can be unpacked using apport-unpack.

1

apport-unpack _tmp_go-build2218735647_b001_exe_crash.0.crash crash_dump.

The unpacked CoreDump can then be analysed with gdb. The other files record some system-related information. (It feels like Ubuntu uses it to allow users to report bugs)

Finally, if the core dump is not generated under the workdir of the process, see if it is in /var/lib/systemd/coredump/. The internet says that systems using systemd store it there, but I haven’t come across it.