TCP State Machine
Connection lifecycle from LISTEN to CLOSED
TCP Connection States
Every TCP connection traverses a well-defined state machine. Understanding these states is essential for diagnosing connection issues — when you run ss or netstat, you're looking at these states.
Server-Side Path (Passive Open)
Client-Side Path (Active Open)
Active Close (Initiator of Close)
Passive Close (Peer of Close Initiator)
All 11 TCP States
| State | Side | Description |
|---|---|---|
CLOSED | Both | No connection exists (fictional starting/ending state) |
LISTEN | Server | Waiting for incoming SYN (passive open) |
SYN_SENT | Client | SYN sent, waiting for SYN-ACK |
SYN_RCVD | Server | SYN received, SYN-ACK sent, waiting for final ACK |
ESTABLISHED | Both | Connection open, data flows bidirectionally |
FIN_WAIT_1 | Active closer | FIN sent, waiting for ACK of FIN |
FIN_WAIT_2 | Active closer | FIN ACKed, waiting for peer's FIN |
TIME_WAIT | Active closer | Both FINs exchanged, waiting 2xMSL before fully closing |
CLOSE_WAIT | Passive closer | Received peer's FIN, waiting for application to call close() |
LAST_ACK | Passive closer | FIN sent, waiting for final ACK |
CLOSING | Both | Simultaneous close — both sides sent FIN before receiving peer's FIN (rare) |
4-Way Connection Teardown
TCP close is a 4-segment exchange (can be collapsed to 3 if FIN+ACK are combined). Each direction is shut down independently — TCP is full-duplex, so each side must send its own FIN.
Teardown Sequence
- FIN (client): Client calls
close(). Kernel sends FIN. Client enters FIN_WAIT_1. - ACK (server): Server's kernel ACKs the FIN. Server enters CLOSE_WAIT. Client enters FIN_WAIT_2.
- FIN (server): Server application calls
close(). Kernel sends FIN. Server enters LAST_ACK. - ACK (client): Client ACKs the FIN. Client enters TIME_WAIT. Server enters CLOSED.
Note
In practice: Many implementations combine steps 2 and 3 into a single FIN+ACK segment, making it a 3-segment teardown. This happens when the server application calls close() quickly after receiving the client's FIN.
TIME_WAIT
The most misunderstood TCP state. After the active closer sends its final ACK, it enters TIME_WAIT for 2 x MSL (Maximum Segment Lifetime). On Linux, MSL = 30s, so TIME_WAIT lasts 60 seconds.
Why TIME_WAIT Exists
- Reliable termination: If the final ACK is lost, the peer will retransmit its FIN. The TIME_WAIT state ensures the closing side can re-ACK it.
- Prevent stale segments: Delayed segments from the old connection might arrive after a new connection is established on the same 5-tuple. TIME_WAIT ensures all old segments have expired (TTL exceeded) before the 5-tuple can be reused.
TIME_WAIT Accumulation
On busy servers that initiate many short-lived connections (e.g., a reverse proxy connecting to backends), TIME_WAIT sockets can accumulate in the tens of thousands.
# Count connections by state
$ ss -s
Total: 15423
TCP: 12847 (estab 3201, closed 1024, orphaned 12, timewait 8610)
# Count TIME_WAIT connections per destination
$ ss -tan state time-wait | awk '{print $4}' | sort | uniq -c | sort -rn | head
3412 10.0.1.5:8080
2891 10.0.1.6:8080
1204 10.0.2.3:443
1103 10.0.2.4:443
Mitigations
| Approach | How | Trade-offs |
|---|---|---|
SO_REUSEADDR |
Socket option — allows bind() to a local address in TIME_WAIT |
Safe, universally used for server sockets |
tcp_tw_reuse |
sysctl net.ipv4.tcp_tw_reuse=1 |
Allows reusing TIME_WAIT sockets for outgoing connections (client-side only). Safe with timestamps enabled. |
| Connection pooling | Keep-alive / persistent connections, HTTP/2 multiplexing | Best solution — avoid the problem entirely. Fewer connections = fewer TIME_WAITs. |
| Let the peer close first | Architecture change — make the server the passive closer | TIME_WAIT lands on the side with more ephemeral ports (the client). |
Warning
Never use tcp_tw_recycle: This sysctl was removed from Linux kernel 4.12 because it breaks connections from clients behind NAT (which share source IPs but have different timestamp clocks). If you see it in old tuning guides, ignore it.
CLOSE_WAIT Accumulation
While TIME_WAIT is normal, CLOSE_WAIT accumulation is always a bug. It means the remote peer has sent a FIN (closed its side), but your application has not called close() on the socket.
Common Causes
- Resource leak: Application reads EOF from the socket but never closes it (missing
close()/defer conn.Close()) - Blocked thread: Application is stuck processing and never returns to its read loop to notice the FIN
- Connection pool bug: Connection returned to pool after peer closed it, pool doesn't validate liveness
# Find CLOSE_WAIT sockets and the owning process
$ ss -tanp state close-wait
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 10.0.0.1:44832 10.0.0.5:443 users:(("myapp",pid=12345,fd=47))
0 0 10.0.0.1:44833 10.0.0.5:443 users:(("myapp",pid=12345,fd=48))
0 0 10.0.0.1:44834 10.0.0.5:443 users:(("myapp",pid=12345,fd=49))
# Count open file descriptors for the suspect process
$ ls -la /proc/12345/fd | wc -l
4096
Tip
Diagnosis workflow: If you see CLOSE_WAIT growing over time, identify the PID with ss -tanp state close-wait, then examine the application code for socket leaks. Check /proc/PID/fd count — if it keeps growing, you have a file descriptor leak.
Half-Open Connections
A half-open connection occurs when one side believes the connection is ESTABLISHED but the other side has no knowledge of it (crashed, rebooted, or network partition).
How Half-Open Connections Happen
- Peer crashes without sending FIN: Process kill -9, kernel panic, power failure — no graceful close
- Network partition: Firewall rule change, route withdrawal, cable unplugged
- NAT timeout: Stateful NAT/firewall drops the mapping after idle timeout, but both endpoints still think connection is alive
Detection Mechanisms
| Mechanism | How It Works | Configuration |
|---|---|---|
| TCP keepalive | After idle period, send probe segments. If no response after retries, declare connection dead. | tcp_keepalive_time=7200 (2h default!), tcp_keepalive_intvl=75, tcp_keepalive_probes=9 |
| Application-level heartbeat | Application protocol sends periodic pings (gRPC keepalive, WebSocket ping/pong) | Much faster detection — typically 10-30 seconds |
| RST on data send | If the surviving side sends data, the rebooted peer responds with RST (it has no record of the connection) | Only works when the surviving side actively sends data |
Warning
The 2-hour trap: Default TCP keepalive is 7200 seconds (2 hours) before the first probe. In K8s environments with pod restarts, service mesh proxies, and cloud load balancers, this is far too long. Always configure application-level keepalives or tune the kernel settings.
# Tune TCP keepalive for K8s/cloud workloads
$ sysctl -w net.ipv4.tcp_keepalive_time=60
$ sysctl -w net.ipv4.tcp_keepalive_intvl=10
$ sysctl -w net.ipv4.tcp_keepalive_probes=6
# Total detection time: 60 + (10 * 6) = 120 seconds
SYN Floods & SYN Cookies
A SYN flood is a classic DoS attack that exploits the TCP handshake. The attacker sends a barrage of SYN segments with spoofed source IPs. The server allocates resources for each half-open connection in the SYN queue and sends SYN-ACKs — but the ACKs never come.
SYN Cookies Defense
SYN cookies (RFC 4987) eliminate the need to store state for half-open connections. Instead of allocating a SYN queue entry, the server encodes the connection state into the ISN itself:
- The server's ISN is a cryptographic hash of (source IP, source port, dest IP, dest port, timestamp, secret key)
- No state is stored — if the ACK comes back, the server reconstructs the connection info from the ack number
- Trade-off: TCP options from the SYN (MSS, window scale, SACK) are lost (partially mitigated by encoding MSS in the hash)
# Check if SYN cookies are enabled (1 = enabled when SYN queue overflows)
$ sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1
# Check SYN queue overflow counter
$ netstat -s | grep "SYNs to LISTEN"
12847 SYNs to LISTEN sockets dropped
# Increase SYN backlog
$ sysctl -w net.ipv4.tcp_max_syn_backlog=4096
Tip
Modern Linux: SYN cookies are enabled by default and activate automatically when the SYN queue overflows. Under normal operation, the kernel uses the standard SYN queue. Under attack, it seamlessly falls back to SYN cookies. You generally don't need to touch this.
Inspecting TCP States with ss
ss (socket statistics) is the modern replacement for netstat. It's faster (reads directly from kernel netlink) and more feature-rich.
# All TCP connections with state
$ ss -tan
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 4096 0.0.0.0:443 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:80 0.0.0.0:*
ESTAB 0 0 10.0.0.1:443 10.0.0.5:52341
TIME-WAIT 0 0 10.0.0.1:443 10.0.0.6:48912
CLOSE-WAIT 0 0 10.0.0.1:44832 10.0.1.5:8080
# Filter by state
$ ss -tan state established
$ ss -tan state time-wait
$ ss -tan state close-wait
# Show process info (requires root)
$ ss -tanp state listening
# Show extended TCP info (cwnd, rtt, etc.)
$ ss -ti dst 10.0.0.5
# Summary of all socket states
$ ss -s
ss Key Flags
| Flag | Meaning |
|---|---|
-t | TCP sockets only |
-u | UDP sockets only |
-a | All states (including LISTEN and TIME-WAIT) |
-n | Numeric output (no DNS resolution, faster) |
-l | Listening sockets only |
-p | Show process using the socket |
-i | Show TCP internal info (cwnd, rtt, mss, etc.) |
-s | Summary statistics |
Note
Recv-Q / Send-Q meaning changes by state:
For LISTEN sockets, Recv-Q = number of pending connections in the accept queue, Send-Q = backlog size.
For ESTABLISHED sockets, Recv-Q = bytes in receive buffer not yet read by app, Send-Q = bytes sent but not yet ACKed.
