error starting container in Kubernetes but not with docker-compose-CodePudding

My container runs fine using docker-compose. But once I apply my deployment on Kubernetes it fails every time on the same error. No matter what I try. I'm stuck and would love some input/help.

I don't know how to debug this. A bit new at Kubernetes. So If someone can guide me, I can try to debug it.

Error stack trace is the following:

Running migration with alembic
INFO  [src.core.config] value for BACKEND_CORS_ORIGINS= ['http://localhost:3000', 'http://localhost:8001']
Traceback (most recent call last):
  File "pydantic/env_settings.py", line 197, in pydantic.env_settings.EnvSettingsSource.__call__
  File "pydantic/env_settings.py", line 131, in pydantic.env_settings.Config.parse_env_var
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())   
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):   
  File "/usr/local/bin/alembic", line 8, in <module>
    sys.exit(main())   
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 590, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 584, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 561, in run_cmd
    fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "alembic/env.py", line 22, in <module>
    from src.core.config import settings
  File "/app/src/core/config.py", line 72, in <module>
    settings = Settings()
  File "pydantic/env_settings.py", line 40, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/env_settings.py", line 75, in pydantic.env_settings.BaseSettings._build_values
  File "pydantic/env_settings.py", line 200, in pydantic.env_settings.EnvSettingsSource.__call__
pydantic.env_settings.SettingsError: error parsing env var "BACKEND_CORS_ORIGINS"
Alembic migration done
[2022-11-15 21:25:36  0000] [1] [INFO] Starting gunicorn 20.1.0
[2022-11-15 21:25:36  0000] [1] [INFO] Listening at: http://0.0.0.0:8001 (1)
[2022-11-15 21:25:36  0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2022-11-15 21:25:36  0000] [23] [INFO] Booting worker with pid: 23
[2022-11-15 21:25:37  0000] [24] [INFO] Booting worker with pid: 24
[2022-11-15 21:25:37  0000] [25] [INFO] Booting worker with pid: 25
[2022-11-15 21:25:40  0000] [23] [ERROR] Exception in worker process

Now how it is set up.

My docker-compose file builds using a Dockerfile. The end of the Dockerfile is as follow. There is an ENTRYPOINT before the CMD instruction

ENTRYPOINT ["./docker-entrypoint.sh"]
CMD ["./run.sh"]

docker-entrypoint.sh

if [ -n "$DB_SERVER" ]; then
  ./wait-for-it.sh "$DB_SERVER:${DB_PORT:-3306}"
fi

# Run the main container command.
exec "$@"

This works because I can see the logs from the 'wait-for-it' script

run.sh (where the error comes from)

export APP_MODULE=${APP_MODULE-src.main:app}
export HOST=${HOST:-0.0.0.0}
export PORT=${PORT:-8001}
export BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS:-'http://localhost:8001'}

echo "Running migration with alembic"
# Run migrations
alembic upgrade head
echo "Alembic migration done"

exec gunicorn -b $HOST:$PORT "$APP_MODULE" --reload --workers=3 --timeout 0 -k uvicorn.workers.UvicornWorker

I've set a default value to BACKEND_CORS_ORIGINS just in case.

In the stack trace above, at the 2nd line you can see a log. This is inside the pydantic validator for BACKEND_CORS_ORIGINS. Here is part of my config.py file:

class Settings(BaseSettings):
  BACKEND_CORS_ORIGINS: List[AnyHttpUrl] = ["http://localhost:3000", "http://localhost:8001"]

  @validator("BACKEND_CORS_ORIGINS", pre=True, allow_reuse=True)
      def assemble_cors_origins(cls, value: Union[str, List[str]]) -> Union[List[str], str]:
          logger.info(f"value for BACKEND_CORS_ORIGINS= {value}, type of BACKEND_CORS_ORIGINS= {type(value)}")
          backend_cors_origins = None
          if isinstance(value, str) and not value.startswith("["):
              backend_cors_origins = [i.strip() for i in value.split(",")]
          elif isinstance(value, (list, str)):
              backend_cors_origins = value
          logger.info(f"value for BACKEND_CORS_ORIGINS= {backend_cors_origins}, type of BACKEND_CORS_ORIGINS= {type(backend_cors_origins)}")
          if backend_cors_origins:
              return backend_cors_origins
          raise ValueError(value)

For some reasons, I am not seeing the last part of the log where I want to see the type.

And to finish, here is my Deployment.yml file for Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: api-name
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-name
  template:
    metadata:
      labels:
        app: api-name
    spec:
      imagePullSecrets:
        - name: secret_name
      containers:
      - name: api-name
        image: [private_registry]/api
        env:
        - name: MARIADB_USER
          value: 'db_user'
        - name: MARIADB_PASSWORD
          value: 'password'
        - name: MARIADB_SERVER
          value: 'IP_DB_SERVER'
        - name: MARIADB_DATABASE
          value: 'db_name'
        - name: MARIADB_PORT
          value: '1234'
        - name: BACKEND_CORS_ORIGINS
          value: 'http://localhost:8001'
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
        ports:
        - containerPort: 8001
---
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: svc-api
spec:
  type: NodePort
  selector:
    app: api-name
  ports:
  - port: 8001
    targetPort: 8001

Can someone help or point my in the right direction? Been trying everything I know to resolve this without any luck.

Maybe I need more config files for Kubernetes?

If I remove the alembic command in the run.sh script, I'm still getting the same error, but without the log inside the @validator

CodePudding user response：

The issue seems to come from this line of export

BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS:-'http://localhost:8001'}

It seems to expect a list of URLs so maybe you can change the code

 - name: BACKEND_CORS_ORIGINS 
   value: 
     - 'localhost:8001'

in Deployment.yml