The company’s internal golang middleware reported a UDP connection exception log, the problem is obvious, the service on the other side down. Restart it and it will be fine.
But the question I’m wondering is how does udp detect when the other side is down?
The udp protocol has neither three handshakes nor TCP status control messages, so how can we determine if the UDP port on the other side is open?
By grabbing the packet, we can find that when the server’s port is not open, the server’s system returns
icmp ECONNREFUSED message to the client, indicating that the connection is abnormal.
When the client system parses the message, it finds the corresponding socket by the five-tuple, and errno returns an exception error, if the client is waiting, it wakes up and sets the error status.
(above is icmp under udp exception, below is normal icmp)
When the UDP connection is abnormal, you can use the tcpdmp utility to specify the ICMP protocol to catch the abnormal message, after all, the other side is ECONNREFUSED via icmp.
Use tcpdump to capture packets
First find a host that can ping through, then use nc to simulate a udp client to request a port that does not exist, and
Connection refused .
The packet capture information is as follows:
Also note that telnet does not support udp, only tcp, so it is recommended to use nc to probe udp.
Testing of various cases
Summary of cases:
- When udp client connects when ip cannot be connected, it usually shows success.
- When the udp server program is closed, but the system still exists, the other system will `icmp ECONNREFUSE error.
- When the other side has iptables udp port drop, the client will also usually show success.
If the IP cannot be connected:
Also, to be clear again udp does not have status messages like tcp, so simply grabbing packets for UDP will not see any abnormal information.
So why does NC UDP command show success when the IP is not connected ?
logic of netcat nc udp
Why does it return a successful connection when the ip is not connected or the message is DROPed ????
Because nc’s default detection logic is simple, as long as no
icmp ECONNREFUSED exception message is received within 2 seconds, then the UDP connection is considered successful. 😅
Here is how the nc udp command is executed.
A UDP client written in golang/ python will not actually report an error when sending a UDP message to an unreachable address, it will usually assume that it was sent successfully.
Still, UDP doesn’t have the handshake step that TCP has, so if you can’t get a response to a TCP syn, the stack will try 6 times with time avoidance, and the kernel will give you an errno value when you don’t get a response after 6 attempts.
UDP connection information
On the client host, you can see the UDP quintuplet connection information via ss lsof netstat.
UDP connection information is usually not visible on the server side, only udp listen information !!!
Client re-instantiation problem ?
When the client and server are connected and the server restarts manually, the client does not need to re-instantiate the connection again and can continue to send data, and when the server starts again, it can still receive messages from the client.
udp has no handshake process, and its udp connect() only creates the socket information locally. The udp quintet of sockets is not visible on the server side using netstat.
Golang test code
When the machine on the udp server side can be connected and there is no exception, the client will usually show success. However, when there is an exception, the following situation will occur:
- When the ip address is not available, the udp client will usually show success when connecting.
- When the udp server program is closed, but the system still exists, the other system returns an error via
icmp ECONNREFUSE, and the client will report an error.
- The client will also show success when the other side has iptables udp port drop operation.
- When the client and server share data, the UDP client cannot sense the shutdown status immediately when the service process hangs, and the client can only sense that the other side is down when the other side responds with an
icmp ECONNREFUSEexception message when sending data again.