We know that, in general, TCP/UDP ports can only be bound to one socket. When we try to listen to a port that is already listened to by another process, the
bind call fails and
errno is set to 98 EADDRINUSE. This is also known as port occupation.
But can a port be listened to by only one process? Obviously not. For example, we can bind a socket and then fork it, so that both child processes are listening on the same port. This is how Nginx does it, with all its worker processes listening on the same port. We can also do the same thing by using a UNIX domain socket to pass a file and “sending” an fd to another process.
In fact, according to TCP/IP standards, ports themselves are allowed to be multiplexed. The essence of port binding is that when the system receives a TCP message or UDP datagram, it can find the corresponding process based on the port field in its header and pass the data to the corresponding process. In addition, for broadcast and multicast, port multiplexing is necessary because they are inherently multi-delivery.
In Linux, we can call
setsockopt to set the
SO_REUSEPORT options to 1 to enable address and port multiplexing.
It’s a little trickier for Go, because Go does all the
listen() steps at once when calling
net.Listen. So we have to use
net.ListenConfig to set up a callback function to control the intermediate process. After getting the raw file descriptor in the callback function, we can call
syscall.SetsockoptInt to set the socket options, which is similar to the original
setsockopt system call.
What does port multiplexing do? According to the TCP/IP standard, for unicast datagrams, if port multiplexing exists, it can only be delivered to one of the processes (as opposed to multicast and broadcast, where it is delivered to all processes). We can make the server more efficient by having multiple processes listening on the same port, allowing them to receive and process data in a preemptive manner. This is how Nginx does it.
The above is an iterative TCP server, which can only handle one connection at a time. But if we enable port multiplexing, if we start N such servers at the same time, they can handle N connections at the same time, and the process is preemptive.
In addition, we can listen to the same ports, with different IP addresses, to process different datagrams at the same time. For example, we can create two goroutines, one listening to 127.0.0.1:1234 and the other listening to 0.0.0.0:1234, to handle different sources differently.
In the above example it is possible to distribute to different goroutines based on different destination IP addresses.