Network Topology

1

[JMeter] -> [ Nginx ] ----> [ Web Server ]

Phenomenon

During the stress test, there were a large number of TIME_WAIT connections on the Web Server, coming from Nginx.

1
2
3

$ netstat -antpl | awk '{print $5, $6}' | sed 's/:[[:print:]]* /\t/g' | sort | uniq -c | sort -rn
  30020 TIME_WAIT <nginx-ip>
     ...

TIME_WAIT is an intermediate state during the 4 waves of the TCP protocol, which all disappear after a while.

Analysis

There are two doubts here.

  1. the number of connections in the TIME_WAIT state exceeds the number of concurrent JMeter connections (500).
  2. the JMeter scripts have Keep-Alive enabled.

This phenomenon looks like the Nginx -> Web Server connection is not kept Alive, causing the connection to be re-established and then disconnected for each request.

Check the Nginx configuration file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

upstream web_server {
    server <ip1>:8080 max_fails=20;
    keepalive 500;
}

map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      close;
}

server {
    listen 80;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://web_server;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 900s;
        proxy_buffering off;
    }
}

As you can see, the keepalive directive has been configured, but it does not appear to be in effect.

However, a look at the documentation for the keepalive directive shows that it needs to be used with the Connection request header set to "".

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

upstream http_backend {
    ...
    keepalive 16;
}

server {
    ...
    location /http/ {
        ...
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        ...
    }
}

And the actual Connection looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      close;
}

server {
    listen 80;
    location / {
        ...
        proxy_set_header Connection $connection_upgrade;
        ...
    }
}

Why do you configure it this way? This is a copy of the official blog of Nginx about proxy Websocket.

According to the above configuration, the value of Connection is actually close, which is why every request disconnects, and then a new request creates a connection, resulting in a lot of TIME_WAIT TCP connections.

Conclusion

The problem is solved by changing the map directive to the following.

1
2
3
4

map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      '';
}