For learning purpose I have created two simple C programms to simulate the following sockets scenario: a client in the loop sends 256 bytes to the server(send
function with MSG_DONTWAIT
flag) but the server doesn't read the data (recv
function). For simplicity the server doesn't send any other data to the client. I would like to test when send
method will return an error. I use linux ss -nmtp
command to see what happens with socket buffers. First the server receive buffer (Recv-Q
) and skmem r
parameter is growing:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 768 0 127.0.0.1:9002 127.0.0.1:34246 users:(("tcpServer",pid=16455,fd=4))
skmem:(r2048,rb131072,t0,tb2626560,f2048,w0,o0,bl0,d0)
ESTAB 0 0 127.0.0.1:34246 127.0.0.1:9002 users:(("tcpClient",pid=16482,fd=3))
skmem:(r0,rb131072,t0,tb2626560,f0,w0,o0,bl0,d0)
When the server Recv-Q
reaches value 98816
the client Send-Q
and skmem w
start growing. When the client send function returns the error 11 (Resource temporarily unavailable) the buffers sizes looks this way:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 98816 0 127.0.0.1:9002 127.0.0.1:49744 users:(("tcpServer",pid=21644,fd=4))
skmem:(r128256,rb131072,t0,tb2626560,f2816,w0,o0,bl0,d16)
ESTAB 0 2553798 127.0.0.1:49744 127.0.0.1:9002 users:(("tcpClient",pid=22042,fd=3))
skmem:(r0,rb131072,t0,tb2626560,f570,w2653638,o0,bl0,d0)
Does skmem rb
refer to the max receive buffer size and skmem tb
refer to the max send buffer size ? If so, how to explain difference between Recv-Q
and skmem rb
and Send-Q
and skmem tb
?
Edit
skmem rb
is the same as the default size returned by the following command:
sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096 131072 6291456
but skmem tb
is different than the default size returned by this command:
sysctl net.ipv4.tcp_wmem
net.ipv4.tcp_wmem = 4096 16384 4194304
How to explain it ?
CodePudding user response:
There are a lot of factors all coming together in your test. I'll try to walk through them. Much of this is documented in the man page for ss
and tcp
.
- Send-Q is the amount of memory currently devoted to buffering outbound data on that socket.
- Recv-Q is the amount currently buffering inbound data on that socket.
- Skmem r is the total allocated memory for received in-flight packets.
- Skmem w is the total allocated memory for sent in-flight packets.
- net.ipv4.tcp_mem contains the low, pressure, and high page count limits on memory for the TCP stack. The default page size is 4,096 so to arrive at bytes you will need to multiply these by the page size.
There are a lot of factors here, but ultimate the memory usage from all TCP sockets combined can not exceed tcp_mem's high setting. If you are currently exceeding the pressure setting, certain limitations may be imposed on connections that are memory heavy. In your case, this test would qualify for those.
This test isn't usually compatible with most real world scenarios, as applications will generally back off long before they receive resource temporarily unavailable.