POST requests get refused after a while on Kubernetes Pod-CodePudding

I want to send data from a sensor written in Python to a Go http Server which are deployed with Kubernetes (k3s) on two Raspberry Pi's. The sensor will read every minute the temperatur and luminosity and send the data as a json with a timestamp to the server. At first when I run the setup it works, but after a while the sensor gets a ConnectionRefusedError: [Errno 111] Connection refused Error in its POST request. However after a while it will continue to work normal until it will break again. I do not know what causes this, since it works part-time. It will just suddenly refuse the connection.

When I use kubectl describe pod weather-sensor-5b88dd65d8-m8zn2 I get under Events:

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  3m16s (x3684 over 18h)  kubelet  Back-off restarting failed container

running kubectl logs weather-sensor-5b88dd65d8-m8zn2 says:

send this data:  {"time": 1635531114, "temp": "23.25", "lux": "254"}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): weather-server:8080
DEBUG:urllib3.connectionpool:http://weather-server:8080 "POST /weather HTTP/1.1" 200 0
send this data:  {"time": 1635531175, "temp": "23.25", "lux": "252"}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): weather-server:8080
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The sensor will then continue to try to connect to the server until it gets a MaxRetryError. Then Kubernetes will terminate the pod because of CrashLoopBackOff

On the sensor I use this url for the post request: URL = "http://weather-server:8080/weather"

In the logs on the server side, I haven't seen anything unusual except that it only gets data erratically.

Relevant Python code:

def create_data(temp, lux):
    weather = {
        'time': int(time.time()),
        'temp': temp,
        'lux': lux
    }
    return json.dumps(weather)


def send_data(data):
    try:
        headers = {'Content-Type': 'application/json'}
        requests.post(url=URL, data=data, headers=headers)
    except ConnectionError as e:
        print(e)

Here my yml files:

sensor_deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-sensor
  labels:
    app: weather
spec:
  replicas: 1
  selector:
    matchLabels:
      app: weather
  template:
    metadata:
      labels:
        app: weather
    spec:
      containers:
      - name: weather-sensor
        image: weather-sensor:pi-1.14
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true

server_deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-server
  labels:
    app: weather
spec:
  replicas: 2
  selector:
    matchLabels:
      app: weather
  template:
    metadata:
      labels:
        app: weather
    spec:
      containers:
      - name: weather-server
        image: weather-server:pi-1.15
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

server_service.yml

apiVersion: v1
kind: Service
metadata:
  name: weather-server
  labels:
    app: weather
spec:
  type: NodePort
  selector:
    app: weather
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080

Any ideas?

CodePudding user response：

I think the problem is the weather server (k8s service) is broken.

This is because the selector is checking for pods with label app=weather which includes both server pods and sensor pods.

If a sensor tries to send data (through the k8s service) to another sensor pod, then it will result in the error because the sensor does not listen for HTTP requests.

To fix it, ensure that the app label is unique for each pod type. For example, weather-server has app=weather-server and weather-sensor has app=weather-sensor.

server_deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-server
  labels:
    app: weather-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: weather-server
  template:
    metadata:
      labels:
        app: weather-server
    spec:
      containers:
      - name: weather-server
        image: weather-server:pi-1.15
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

server_service.yml

apiVersion: v1
kind: Service
metadata:
  name: weather-server
  labels:
    app: weather-server
spec:
  type: NodePort
  selector:
    app: weather-server
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080