1. Data reception process
The process of external data from a computer system is relatively long from the time it enters the NIC to the time it is finally received by the application. The approximate flow is as follows.
- The NIC receives the data, copies it to the kernel space through the DMA controller, and at the same time initiates a hard interrupt to the CPU.
- The CPU receives the hard interrupt and simply processes it, then hands it over to the ksoftirqd process, a process called soft interrupt.
- The ksoftirqd process is used to interact with IO multiplexing, such as select, epoll, etc. It will trigger poll().
- After the copy is done by IO model (epoll for example), the data will be added to one of the fd in the ready list.
- By callback (epoll) as an example) the corresponding process is woken up to access the data.
Regarding the ksoftirqd process, it is used to handle soft interrupts and call the registered poll method to start receiving packets.
2. The overall soft and hard interaction process of epoll
The interaction logic between the kernel and external devices and applications is roughly as follows.
- The OS will create a call to
epoll_createat startup to create an epoll management object.
- The application will bind and listen to layer 7 data via http/rpc etc. at startup.
- Register listen events with the kernel and call
epoll_waitto block and wait for the data to be ready.
- When the data is ready it is copied from kernel to user space and wakes up for execution.
3. Initialize epfd
atomic to implement the singleton pattern to prevent being initialized multiple times. The specific instantiation is in
Due to the Cow mechanism, before the child process executes, it has the same data space, stack and fd list as its parent process. When the child process executes new code, the fd of the parent process will also be overwritten, which will lead to the inability to maintain the fd of the parent process, so the child process needs to turn off the useless fd first and then execute the corresponding code.
By IO multiplexing, we mean that all threads multiplex a connection.
netpoll plays a top-down role.
r and w are the lowest level read and write fd’s. They are also the fd’s to be reused, after which all fd’s registered to listen in netpoll are reused for this
netpoll model, all modifications to the
netpoll object are implemented through
netpoll object itself is also a file descriptor. It is also known as
epfd in the code.
Here the readable event is bound to the underlying netpollBreakRd, i.e. data coming from external network connections are written to the kernel via netpollBreakRd and then distributed to different fd’s.
4. Bind events to user fd
It’s relatively simple here. You call
epollctl to bind the readable, writable, pending and edge trigger events. This way the corresponding fd can be invoked immediately when ready to read and write data.
5. Unbind event to user fd
epollctl and specify the operation as
6. Listening for events
The listening part is a constant loop for the events registered via
netpoll_ctx, and when the multiplexed connections (
netpollBreakWd) are ready for reading and writing, it wakes up the corresponding event-bound Goroutine.
The main processes are as follows.
initializei.e. get the underlying connections
netpollBreakWr. These two are external read/write connection streams, which are themselves a socket object.
epollmultiplexes these two fd’s.
eventpollfd, through which the read and write events of other sockets are managed.
epfdis the core structure of the
Registered eventsA Goroutine initiates an IO request through a system call, the kernel registers the
epollctlto bind the required events, such as read events, write events, hang events, etc. Then Goroutine enters a blocking state and waits to be woken up.
Listening eventsThe kernel constantly listens for the underlying connection
netpollBreakWr. When the underlying connection is readable,
netpollBreadRdis read and then cached on the kernel data space.
Listen to eventsThe kernel keeps looping according to various events registered by
epfd, and when a registered event appears, such as readable, writable, etc. it wakes up the Goroutine corresponding to the bound event, i.e. the
Goroutine ready, Goroutine receives the notification and starts reading and writing data to
fd. After it finishes, it closes
fd. The kernel removes its bound events from