I've encountered rather strange behavior of my kubernetes cluster (1.18.20, calico 3.14.2): when I attempt to upload two-megabyte JSON file to pod via curl through NodePort service, transmission is interrupted with Recv failure: Connection reset by peer. Traffic capture shows that both client and server receive RST packets from network, but didn't sent them. Binary file of same size uploads successfully, but JSON is rejected regardless of Content-Type specified. File transfer between pods (using similar commands and same file) proceeds smoothly. Upload through ingress (also configured using NodePort) fails too. Size of received fragment is always the same, approximately 850K.

I've used nc -l 80* instead of real service with the same outcome.

Apparently, kube-proxy doesn't like big JSON files.

Is it possible to send big JSON files to pod from external clients, or such a limit is hardcoded?

UPD1

Same behavior for fresh cluster (1.22.0, calico 3.20.0).

UPD2

System rejects not every big JSON payload, but only several percent of user uploads. Payload is specially crafted by client application: first part of multy-volume Zip archive is base64-encoded and encapsulated as JSON file ('{ "data": "..." }'). Size of fragment causing connection break is about 640K.

Looks like error in filtering procedure inside of kube-proxy.

CodePudding user response：

Have you tried to use conntrack to solve your connection issue? It simply installs by the following command:

apt-get install conntrack #[ debian/ubuntu ]
yum install conntrack #[ centos/redhat ] 

You can experiment whatever you would like with the max size, but you can start with the following metrics to check if something happens:

sysctl -w net.netfilter.nf_conntrack_buckets=300
sysctl -w net.netfilter.nf_conntrack_max=1200

Please find more detailed guide here

CodePudding user response：

Unfortunately, the source of the problem was in misconfiguration of IDS/IPS.

Nothing to do with kube-proxy.