Compile and modify Golang source code for debugging
First download and compile
I’m using a centos environment, so I need to install yum -y install gcc first.
Then download the go source code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[root@localhost src]# git clone https://github.com/golang/go.git
#进入到src 目录执行
[root@localhost src]# ./all.bash
[root@localhost src]# ./all.bash
Building Go cmd/dist using /usr/local/go. (go1.15.8 linux/amd64)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
...
##### API check
Go version is "go1.15.10", ignoring -next /data/learn/go/api/next.txt
ALL TESTS PASSED
---
Installed Go for linux/amd64 in /data/learn/go
Installed commands in /data/learn/go/bin
*** You need to add /data/learn/go/bin to your PATH.
|
The compiled go and gofmt are in the bin directory.
1
2
3
|
[root@localhost src]# cd ../bin/
[root@localhost bin]# ls
go gofmt
|
To prevent conflicts between our modified go and the go we installed in the past, create a mygo softlink that points to the modified go.
1
2
3
4
5
|
[root@localhost bin]# mkdir -p ~/mygo/bin
[root@localhost bin]# cd ~/mygo/bin
[root@localhost bin]# ln -sf /data/learn/go/bin/go mygo
|
Finally, add ~/testgo/bin to the PATH.
1
2
3
4
5
|
[root@localhost bin]# vim /etc/profile
export PATH=$PATH:/data/learn/mygo/bin
source /etc/profile
|
Run mygo and check the version.
1
2
|
[root@localhost bin]# mygo version
go version go1.15.10 linux/amd64
|
GODEBUG
When we modify the source code, we can print debugging information with the help of GODEBUG variables.
schedtrace
schedtrace: setting schedtrace=X causes the scheduler to emit a single line to standard
error every X milliseconds, summarizing the scheduler state.
For example, set schedtrace=1000 to print one line of scheduler summary information every one second after the program starts.
1
2
3
4
|
[root@localhost gotest]# GOMAXPROCS=1 GODEBUG=schedtrace=1000 mygo run main.go SCHED 0ms: gomaxprocs=1 idleprocs=0 threads=4 spinningthreads=0 idlethreads=0 runqueue=0 [2]
# command-line-arguments
SCHED 0ms: gomaxprocs=1 idleprocs=0 threads=3 spinningthreads=0 idlethreads=1 runqueue=0 [2]
SCHED 0ms: gomaxprocs=1 idleprocs=0 threads=3 spinningthreads=0 idlethreads=0 runqueue=0 [2]
|
0ms : the number of milliseconds since the start of the program.
gomaxprocs=1: the number of configured processors.
idleprocs=0: number of idle P(processors).
threads=3: number of threads managed during runtime, currently 6 threads.
spinningthreads=0: number of threads preempted by execution.
idlethreads=1: number of idle threads.
runqueue=0: the number of goroutines in the global run queue.
[2]: the number of goroutines in the local run queue, indicating that two are waiting.
scheddetail
scheddetail: setting schedtrace=X and scheddetail=1 causes the scheduler to emit
detailed multiline info every X milliseconds, describing state of the scheduler,
processors, threads and goroutines.
schedtrace and scheddetail together can be set to provide details about processor P, thread M and goroutine G.
Example.
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[root@localhost gotest]# GOMAXPROCS=1 GODEBUG=schedtrace=1000,scheddetail=1 mygo run main.go
SCHED 0ms: gomaxprocs=1 idleprocs=0 threads=4 spinningthreads=0 idlethreads=0 runqueue=0 gcwaiting=0 nmidlelocked=0 stopwait=0 sysmonwait=0
P0: status=0 schedtick=0 syscalltick=0 m=-1 runqsize=2 gfreecnt=0 timerslen=0
M3: p=-1 curg=-1 mallocing=0 throwing=0 preemptoff= locks=0 dying=0 spinning=false blocked=false lockedg=-1
M2: p=-1 curg=-1 mallocing=0 throwing=0 preemptoff= locks=2 dying=0 spinning=false blocked=false lockedg=-1
M1: p=-1 curg=17 mallocing=0 throwing=0 preemptoff= locks=0 dying=0 spinning=false blocked=false lockedg=17
M0: p=-1 curg=-1 mallocing=0 throwing=0 preemptoff= locks=1 dying=0 spinning=false blocked=false lockedg=1
G1: status=1(chan receive) m=-1 lockedm=0
G17: status=6() m=1 lockedm=1
G2: status=1() m=-1 lockedm=-1
G3: status=1() m=-1 lockedm=-1
G4: status=4(GC scavenge wait) m=-1 lockedm=-1
...
|
Let’s start by looking at what G stands for.
- status: the operational status of G.
- m: which M it belongs to.
- lockedm: whether or not there is a locked M.
The operational status of G probably has these meanings.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
const (
// 刚刚被分配并且还没有被初始化
_Gidle = iota // 0
// 没有执行代码，没有栈的所有权，存储在运行队列中
_Grunnable // 1
// 可以执行代码，拥有栈的所有权，被赋予了内核线程 M 和处理器 P
_Grunning // 2
// 正在执行系统调用，拥有栈的所有权，没有执行用户代码，
// 被赋予了内核线程 M 但是不在运行队列上
_Gsyscall // 3
// 由于运行时而被阻塞，没有执行用户代码并且不在运行队列上，
// 但是可能存在于 Channel 的等待队列上
_Gwaiting // 4
// 表示当前goroutine没有被使用，没有执行代码，可能有分配的栈
_Gdead // 6
// 栈正在被拷贝，没有执行代码，不在运行队列上
_Gcopystack // 8
// 由于抢占而被阻塞，没有执行用户代码并且不在运行队列上，等待唤醒
_Gpreempted // 9
// GC 正在扫描栈空间，没有执行代码，可以与其他状态同时存在
_Gscan = 0x1000
...
)
|
What M stands for.
1
|
M0: p=-1 curg=-1 mallocing=0 throwing=0 preemptoff= locks=1 dying=0 spinning=false blocked=false lockedg=1
|
- p: which P to belong to.
- curg: which G is currently in use.
- mallocing: whether memory is being allocated.
- throwing: whether an exception is being thrown.
- preemptoff: keep curg running on this m if it is not equal to the empty string ("").
- runqsize: the number of G’s in the run queue.
- spinning: whether G is being preempted or not.
What P stands for.
1
|
P0: status=0 schedtick=0 syscalltick=0 m=-1 runqsize=2 gfreecnt=0 timerslen=0
|
- status: the running status of P.
- schedtick: the number of times P was scheduled.
- syscalltick: the number of system calls of P.
- m: which M to belong to.
- runqsize: the number of G’s in the run queue.
- gfreecnt: the available Gs (with status Gdead).
The status state of P represents the following.
1
2
3
4
5
6
7
8
9
10
11
12
|
const (
// 表示P没有运行用户代码或者调度器
_Pidle = iota
// 被线程 M 持有，并且正在执行用户代码或者调度器
_Prunning
// 没有执行用户代码，当前线程陷入系统调用
_Psyscall
// 被线程 M 持有，当前处理器由于垃圾回收 STW 被停止
_Pgcstop
// 当前处理器已经不被使用
_Pdead
)
|
Modify the compilation
Let’s say we make a change to the channel and add a print to it.
1
2
3
4
5
6
7
8
|
func makechan(t *chantype, size int) *hchan {
...
if debug.schedtrace > 0 {
print("bearluo makechan: chan=", c, "; elemsize=", elem.size, "; dataqsiz=", size, "\n")
}
...
return c
}
|
Then go to the src directory of go and recompile.
1
2
3
4
5
6
7
8
9
10
|
[root@localhost src]# ./make.bash
Building Go cmd/dist using /usr/local/go. (go1.15.8 linux/amd64)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /data/learn/go
Installed commands in /data/learn/go/bin
|
Write a simple demo
1
2
3
4
5
6
7
8
9
10
11
|
package main
import (
"fmt"
)
func main() {
c := make(chan int, 10)
fmt.Println(c)
}
|
run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[root@localhost gotest]# GODEBUG=schedtrace=1000 mygo run main.go
bearluo makechan: chan=0xc000036070; elemsize=8; dataqsiz=2
SCHED 0ms: gomaxprocs=16 idleprocs=13 threads=6 spinningthreads=1 idlethreads=0 runqueue=0 [1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
bearluo makechan: chan=0xc00010e000; elemsize=1; dataqsiz=0
bearluo makechan: chan=0xc00010e060; elemsize=0; dataqsiz=0
bearluo makechan: chan=0xc00010e180; elemsize=0; dataqsiz=0
bearluo makechan: chan=0xc0006a8000; elemsize=1; dataqsiz=35
bearluo makechan: chan=0xc0003f6660; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000226540; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc0001381e0; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc0005043c0; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc00049c420; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000594300; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000090360; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000220000; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc00075e000; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000138840; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc000226780; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc0003ea420; elemsize=16; dataqsiz=2
bearluo makechan: chan=0xc00049d320; elemsize=16; dataqsiz=1
...
|
Delve Debugging
Currently Go language supports several debuggers, GDB, LLDB and Delve. Only Delve is a debugging tool designed and developed specifically for the Go language. Delve itself is also developed in Go language and provides the same support for Windows platform. In this section, we briefly explain how to debug Go runtime code and assembler based on Delve.
Github: https://github.com/go-delve/delve
Installation.
1
|
go get github.com/go-delve/delve/cmd/dlv
|
First write an example of test.go.
1
2
3
4
5
6
7
8
9
10
11
|
package main
import "fmt"
type A struct {
test string
}
func main() {
a := new(A)
fmt.Println(a)
}
|
Then enter the package directory at the command line and enter the dlv debug command to debug.
1
2
|
PS C:\document\code\test_go\src> dlv debug
Type 'help' for list of commands.
|
Then you can use the break command to set a breakpoint on the main method of the main package
1
2
|
(dlv) break main.main
Breakpoint 1 set at 0x4bd30a for main.main() c:/document/code/test_go/src/test.go:8
|
View all breakpoints that have been set via breakpoints.
1
2
3
4
5
|
(dlv) breakpoints
Breakpoint runtime-fatal-throw at 0x4377e0 for runtime.fatalthrow() c:/software/go/src/runtime/panic.go:1162 (0)
Breakpoint unrecovered-panic at 0x437860 for runtime.fatalpanic() c:/software/go/src/runtime/panic.go:1189 (0)
print runtime.curg._panic.arg
Breakpoint 1 at 0x4bd30a for main.main() c:/document/code/test_go/src/test.go:8 (0)
|
The continue command allows the program to run to the next breakpoint
1
2
3
4
5
6
7
8
9
10
11
12
13
|
(dlv) continue
> main.main() c:/document/code/test_go/src/test.go:8 (hits goroutine(1):1 total:1) (PC: 0x4bd30a)
3: import "fmt"
4:
5: type A struct {
6: test string
7: }
=> 8: func main() {
9: a := new(A)
10: fmt.Println(a)
11: }
12:
13:
|
View the assembly code corresponding to the main function by using the disassemble disassembly command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
(dlv) disassemble
TEXT main.main(SB) C:/document/code/test_go/src/test.go
test.go:8 0x4bd2f0 65488b0c2528000000 mov rcx, qword ptr gs:[0x28]
test.go:8 0x4bd2f9 488b8900000000 mov rcx, qword ptr [rcx]
test.go:8 0x4bd300 483b6110 cmp rsp, qword ptr [rcx+0x10]
test.go:8 0x4bd304 0f8697000000 jbe 0x4bd3a1
=> test.go:8 0x4bd30a* 4883ec78 sub rsp, 0x78
test.go:8 0x4bd30e 48896c2470 mov qword ptr [rsp+0x70], rbp
test.go:8 0x4bd313 488d6c2470 lea rbp, ptr [rsp+0x70]
test.go:9 0x4bd318 488d0581860100 lea rax, ptr [__image_base__+874912]
test.go:9 0x4bd31f 48890424 mov qword ptr [rsp], rax
test.go:9 0x4bd323 e8e800f5ff call $runtime.newobject
test.go:9 0x4bd328 488b442408 mov rax, qword ptr [rsp+0x8]
test.go:9 0x4bd32d 4889442430 mov qword ptr [rsp+0x30], rax
test.go:10 0x4bd332 4889442440 mov qword ptr [rsp+0x40], rax
test.go:10 0x4bd337 0f57c0 xorps xmm0, xmm0
test.go:10 0x4bd33a 0f11442448 movups xmmword ptr [rsp+0x48], xmm0
test.go:10 0x4bd33f 488d442448 lea rax, ptr [rsp+0x48]
test.go:10 0x4bd344 4889442438 mov qword ptr [rsp+0x38], rax
test.go:10 0x4bd349 8400 test byte ptr [rax], al
test.go:10 0x4bd34b 488b4c2440 mov rcx, qword ptr [rsp+0x40]
test.go:10 0x4bd350 488d15099f0000 lea rdx, ptr [__image_base__+815712]
test.go:10 0x4bd357 4889542448 mov qword ptr [rsp+0x48], rdx
test.go:10 0x4bd35c 48894c2450 mov qword ptr [rsp+0x50], rcx
test.go:10 0x4bd361 8400 test byte ptr [rax], al
test.go:10 0x4bd363 eb00 jmp 0x4bd365
test.go:10 0x4bd365 4889442458 mov qword ptr [rsp+0x58], rax
test.go:10 0x4bd36a 48c744246001000000 mov qword ptr [rsp+0x60], 0x1
test.go:10 0x4bd373 48c744246801000000 mov qword ptr [rsp+0x68], 0x1
test.go:10 0x4bd37c 48890424 mov qword ptr [rsp], rax
test.go:10 0x4bd380 48c744240801000000 mov qword ptr [rsp+0x8], 0x1
test.go:10 0x4bd389 48c744241001000000 mov qword ptr [rsp+0x10], 0x1
test.go:10 0x4bd392 e869a0ffff call $fmt.Println
test.go:11 0x4bd397 488b6c2470 mov rbp, qword ptr [rsp+0x70]
test.go:11 0x4bd39c 4883c478 add rsp, 0x78
test.go:11 0x4bd3a0 c3 ret
test.go:8 0x4bd3a1 e82a50faff call $runtime.morestack_noctxt
.:0 0x4bd3a6 e945ffffff jmp $main.main
|
Now we can use the break breakpoint to the call to the runtime.newobject function
1
2
|
(dlv) break runtime.newobject
Breakpoint 2 set at 0x40d426 for runtime.newobject() c:/software/go/src/runtime/malloc.go:1164
|
Type continue to jump to the location of the breakpoint.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
(dlv) continue
> runtime.newobject() c:/software/go/src/runtime/malloc.go:1164 (hits goroutine(1):1 total:1) (PC: 0x40d426)
Warning: debugging optimized function
1159: }
1160:
1161: // implementation of new builtin
1162: // compiler (both frontend and SSA backend) knows the signature
1163: // of this function
=>1164: func newobject(typ *_type) unsafe.Pointer {
1165: return mallocgc(typ.size, typ, true)
1166: }
1167:
1168: //go:linkname reflect_unsafe_New reflect.unsafe_New
1169: func reflect_unsafe_New(typ *_type) unsafe.Pointer {
|
print command to view the data of typ.
1
2
|
(dlv) print typ
*runtime._type {size: 16, ptrdata: 8, hash: 875453117, tflag: tflagUncommon|tflagExtraStar|tflagNamed (7), align: 8, fieldAlign: 8, kind: 25, equal: runtime.strequal, gcdata: *1, str: 5418, ptrToThis: 37472}
|
You can see that the size printed here is 16bytes, because we have a string type field inside the A structure.
Once you get to the mallocgc method, you can view the arguments and local variables of the function by using the args and locals commands.
1
2
3
4
5
6
7
|
(dlv) args
size = (unreadable could not find loclist entry at 0x8b40 for address 0x40ca73)
typ = (*runtime._type)(0x4d59a0)
needzero = true
~r3 = (unreadable empty OP stack)
(dlv) locals
(no locals)
|