Worker unable to connect to the master and invalid args in webport for Locust-CodePudding

I am trying to set up a load test for an endpoint. This is what I have followed so far:

`Dockerfile`

FROM python:3.8

# Add the external tasks directory into /tasks
WORKDIR /src

ADD requirements.txt .
RUN pip install --no-cache-dir --upgrade locust==2.10.1

ADD run.sh .
ADD load_test.py .
ADD load_test.conf .

# Expose the required Locust ports
EXPOSE 5557 5558 8089

# Set script to be executable
RUN chmod 755 run.sh

# Start Locust using LOCUS_OPTS environment variable
CMD ["bash", "run.sh"]

# Modified from:
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/docker/Dockerfile

`run.sh`

#!/bin/bash
LOCUST="locust"
LOCUS_OPTS="--config=load_test.conf"
LOCUST_MODE=${LOCUST_MODE:-standalone}

if [[ "$LOCUST_MODE" = "master" ]]; then
    LOCUS_OPTS="$LOCUS_OPTS --master"
elif [[ "$LOCUST_MODE" = "worker" ]]; then
    LOCUS_OPTS="$LOCUS_OPTS --worker --master-host=$LOCUST_MASTER_HOST"
fi

echo "${LOCUST} ${LOCUS_OPTS}"

$LOCUST $LOCUS_OPTS

# Copied from
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/docker/locust/run.sh

This is how I have written the load test locust script:

import json

from locust import HttpUser, constant, task


class CategorizationUser(HttpUser):
    wait_time = constant(1)

    @task
    def predict(self):
        payload = json.dumps(
            {
                "text": "Face and Body Paint washable Rubies Halloween item 91#385"
            }
        )
        _ = self.client.post("/predict", data=payload)

I am invoking that with a configuration:

locustfile = load_test.py
headless = false
users = 1000
spawn-rate = 1
run-time = 5m
host = IP
html = locust_report.html

So, after building and pushing the Docker image and creating a k8s cluster on GKE, I am deploying it. This is how the deployment.yaml looks like:

# Copied from 
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/kubernetes/templates/deployment.yaml

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: locust-master-deployment
  labels:
    name: locust
    role: master
spec:
  replicas: 1
  selector:
    matchLabels:
      name: locust
      role: master
  template:
    metadata:
      labels:
        name: locust
        role: master
    spec:
      containers:
        - name: locust
          image: gcr.io/PROJECT_ID/IMAGE_URI
          imagePullPolicy: Always
          env:
            - name: LOCUST_MODE
              value: master
            - name: LOCUST_LOG_LEVEL
              value: DEBUG
          ports:
            - name: loc-master-web
              containerPort: 8089
              protocol: TCP
            - name: loc-master-p1
              containerPort: 5557
              protocol: TCP
            - name: loc-master-p2
              containerPort: 5558
              protocol: TCP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: locust-worker-deployment
  labels:
    name: locust
    role: worker
spec:
  replicas: 2
  selector:
    matchLabels:
      name: locust
      role: worker
  template:
    metadata:
      labels:
        name: locust
        role: worker
    spec:
      containers:
        - name: locust
          image: gcr.io/PROJECT_ID/IMAGE_URI
          imagePullPolicy: Always
          env:
            - name: LOCUST_MODE
              value: worker
            - name: LOCUST_MASTER
              value: locust-master-service
            - name: LOCUST_LOG_LEVEL
              value: DEBUG

After deployment, I am exposing the required ports like so:

kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
    --type NodePort \
    --port 5558 \
    --target-port 5558 \
    --name locust-5558

kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
    --type NodePort \
    --port 5557 \
    --target-port 5557 \
    --name locust-5557

kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
    --type LoadBalancer \
    --port 80 \
    --target-port 8089 \
    --name locust-web

The cluster and the nodes provision successfully. But the moment I hit the IP of locust-web, I am getting:

Any suggestions on how to resolve the bug?

CodePudding user response：

Since you are exposing your pods and you are trying to access to them outside the cluster (your web application), you have to port-forward them or add an Ingress in order to access to your locust pods. My first approach will be trying to ping or send requests to your locust pods with a simple port-forward.

More infos about the port forwarding here.

CodePudding user response：

Probably the environment variables set by k8s is colliding with locust’s (LOCUST_WEB_PORT specifically). Change your setup so that no containers are named ”locust”.

See https://github.com/locustio/locust/issues/1226 for a similar issue.