Programmers who are a little bit introduced to the Go language know that the
GOMAXPROCS variable can limit the maximum number of threads that can concurrently run the user-state
Go code operating system, and you can even change the size of the maximum number of threads at program runtime by calling the function
func GOMAXPROCS(n int) int, but when you read the documentation further, or more When you read the documentation further, or go deeper into
Go language development, you will find that the actual number of threads is larger than the number you set, sometimes much larger than the value you set, and even more tragically, even when your concurrent tasks are backed up to no more than a few, the number of threads has not come down, wasting memory space and
CPU scheduling for nothing.
Of course, this problem has been encountered by many people.
Go documentation also indicates that the actual
Thread may not be limited by
GOMAXPROCS, and that the number of threads being
blocked when the
Go code makes a system call is not limited by this variable, as stated in the following documentation.
The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit. This package’s GOMAXPROCS function queries and changes the limit.
If there are a lot of concurrent
blocking system calls,
Go will create a lot of threads, but when the system calls are done, these threads will not be recycled because of the design of the
Go runtime. See
go issue #14592 for a detailed discussion. This
issue is a
2016, it’s been a few years since
Go 1.6 was pushed, and still no one has done anything to try to fix or improve it. Obviously, it’s not an easy fix to work on.
I’ll reorganize it to deepen my understanding of this point.
What is a blocking system call?
So what is a blocking system call? stackoverflow has a Q&A that answers this question very well.
A blocking system call is one that must wait until the action can be completed. read() would be a good example - if no input is ready, it’ll sit there and wait until some is (provided you haven’t set it to non-blocking, of course, in which case it wouldn’t be a blocking system call). Obviously, while one thread is waiting on a blocking system call, another thread can be off doing something else.
A blocking system call is a system call that executes while the caller must wait until it completes.
read() is a good example, if there is no data to read, the caller waits until some data is available (in case you don’t set it to
Go read data from network
I/O take up a system thread for each read
goroutine? No way!
netpoller to handle network reads and writes, and it uses
kqueue(BSD, Darwin), and
poll network I/O state. Once a connection is accepted, the file descriptor of the connection is set to
non-blocking, which also means that once there is no data in the connection,
read data from it is not blocked, but a specific error is returned, so the
Go standard library network read/write does not generate a lot of threads, unless you set
GOMAXPROCS very large or set the underlying network connection file descriptor back to
cgo or some other blocking system call may cause a large number of threads to be added and not reclaimed, as in the following example.
A simple test for dramatic thread count increase
Let me give you a simple example so you can see the large number of threads that are created.
Go provides two ways to query domain names, the
CGO way or the pure
Go way, such as
LookupAddr in the
net library, these functions will be indirectly or directly related to the domain name program, such as the above example using
LookupHost, using different ways to generate different threads in the case of concurrency.
For example, with a pure
Go approach, the program would have
10 threads at exit time.
cgo approach, the program exits with dozens, if not hundreds, of threads at the time of.
Infinite increase? No way!
Go runtime will not recycle threads, but will reuse them when needed. But you simply don’t need them if you create a lot of threads, and the theoretical value of keeping a small number of threads for reuse is fine.
If the program is not designed properly, it can result in a large number of idle threads. If you call similar
blocking system calls or
CGO code in the http handler, or if you call similar code on the microservices server, it is possible to create a “thread leak” when the client has high concurrency.
However, there is no limit to the number of threads that can be created. For one thing, each thread takes up a certain amount of memory resources, and a large number of threads can lead to memory exhaustion, and the
Go runtime actually has a display for the number of threads created at runtime, which is
10000 by default.
You can set this using the
debug.SetMaxThreads function. For example, you can set the maximum number of threads to
100 in the above example:
If you run the above program again, it crashes:
Reduction of threads
There is also a way to kill a thread using LockOSThread in the official issue
LockOSThread function binds the current
goroutine to the current system thread. This
goroutine is always executed in this thread, and no other
goroutine is executed in this thread. Only after this
goroutine has called the
UnlockOSThread function the same number of times, will it be unbound.
goroutine exits without
unlocking the thread, then the thread will be terminated. We can use this feature to kill the thread. We can start a
LockOSThread to occupy a thread, although there are currently many free threads, so we can just reuse one,
goroutine does not call
UnlockOSThread when it exits, which also causes the thread to be terminated.
Of course there are concerns provided in the official
issue that killing an idle thread could potentially result in a child process that would receive the
You can extend this method by providing
Kill(n int) which can terminate multiple threads, but of course the principles are similar. From a practical point of view, you can start a guarded
goroutine, which will recycle some threads when the number of threads exceeds a certain threshold, or provide an interface to manually call some
API to terminate some threads, which is an available method before the official solution to this problem.