Eventual failure: kubectl exec fails with "operation not permitted: unkown"-CodePudding

I have some Pods that are running some Python programs. Initially I'm able to execute simple commands into the Pods. However after some time (maybe hours?) I start to get the following error:

$ kubectl exec -it mypod -- bash
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "37a9f1042841590e48e1869f8b0ca13e64df02d25458783e74d8e8f2e33ad398": OCI runtime exec failed: exec failed: unable to start container pr
ocess: open /dev/pts/0: operation not permitted: unknown

If I restart the Pods, then this clears the condition. However, I'd like to figure out why this happening to avoid having to restart Pods each time.

The Pods are running a simple Python script, and the Python program is still running as normal (kubectl logs shows what I expect).

Also, I'm running K3s for Kubernetes across 4 nodes (1 master, 3 workers). I noticed all Pods running on certain nodes started to experience this issue. For example, initially I found all Pods running on worker2 and worker3 had this issue (but all Pods on worker1 did not). Eventually all Pods across all worker nodes start to have this problem. So it appears to be related to a condition on the node that is preventing exec from running. However restarting the Pods resets the condition.

As far as I can tell, the containers are running fine in containerd. I can log into the nodes and containerd shows the containers are running, I can check logs, etc...

What else should I check?
Why would the ability to exec stop working? (but containers are still running)

CodePudding user response：

There is a couple of GitHub issues one or another from the middle of august. They said it was an SELinux issue and fixed in the runc v1.1.4. You should check your runc version and when it is below the mentioned version then update it.

Otherwise, you can disable SELinux when you aren't working in production:

setenforce 0

or when you want some more sophisticated solution, try this: https://github.com/moby/moby/issues/43969#issuecomment-1217629129