How to fix server load caused by many CLOSE_WAIT connections on CentOS Linux

Sometimes, you might face a problem when you have a huge load on a server caused by an opened TCP connections which stay for a long time. Actually it happens when the application does not send proper close signal to the opened socket connection. It might be faulty issue or made intentionally.

Large amount of such unclosed connections might load your server and make it useless.

To check the current state of the connections status, use the next command:

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

This will give you summary info about connections grouped by connection type:

      1 established
      1 FIN_WAIT1
      1 Foreign
      5 SYN_RECV
     16 FIN_WAIT2
     21 LISTEN
     80 TIME_WAIT
    113 ESTABLISHED
    983 CLOSE_WAIT

I actually prefer get summary info grouped by IP and connection type, to check the suspicious/violating IP-es:

netstat -nat | awk '{print $5 " " $6}' | sort | uniq -c | awk '$1 > 10 {print $1 " " $2 " " $3}' | sort -n

In this case you’ll see top IP addresses with highest amount of connections (e.g. above 10) grouped by connection type:

11 82.74.65.12:58572 LISTEN
35 92.101.98.101:60706 CLOSE_WAIT
60 105.104.58.35:62211 CLOSE_WAIT

By default, maximum timeout between first and last packet that tcp socket connection can listen – in other words – keeps connection alive – is 7200 seconds. This default parameters could be checked by the next command:

# sysctl -a | grep tcp_keepalive

net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200

As a best practice, it’s recommended to reduce that times which could be accomplished by next two approaches:

1. Modify /etc/sysctl.conf

# add this lines
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6

# reload new settings
sysctl -p

2. Modify setting files

# echo 60 > /proc/sys/net/ipv4/tcp_keepalive_time

# echo 10 > /proc/sys/net/ipv4/tcp_keepalive_intvl

# echo 6 > /proc/sys/net/ipv4/tcp_keepalive_probes

@source:
https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
https://webhostinggeeks.com/howto/configure-linux-tcp-keepalive-setting/

Leave a Reply Cancel reply