ModuleNotFoundError when starting celery in Django project-CodePudding

I have a Django project structure like this (not showing all the files/folders):

myproj/
├── __init__.py
├── config/
│   ├── __init__.py    <-- imports `celery_app` as per docs
│   ├── settings/
│   │   ├── __init__.py
│   │   └── base.py
│   ├── celery.py      <-- defines `app = Celery("myproj")`
│   └── wsgi.py     
├── myapp1/
└── myapp2/

I have a docker-compose.yml that sets up web and worker services, and the worker service is something like:

  worker:
    build:
      context: "."
    target: "app"
    command: celery -A config worker -l info
    entrypoint: []

But whatever I try for the -A when starting celery I get ModuleNotFound. I've tried config, myproj.config and myproj. I've tried adding --workdir=/path/to/myproj, but no joy.

UPDATE With -A config I get this:

oohdir-worker-1    | Usage: celery [OPTIONS] COMMAND [ARGS]...
oohdir-worker-1    | Try 'celery --help' for help.
oohdir-worker-1    | 
oohdir-worker-1    | Error: Invalid value for '-A' / '--app': 
oohdir-worker-1    | Unable to load celery application.
oohdir-worker-1    | The module config was not found.
oohdir-worker-1 exited with code 2

But with -A myproj.config I get this:

myproj-worker-1    | Usage: celery [OPTIONS] COMMAND [ARGS]...
myproj-worker-1    | Try 'celery --help' for help.
myproj-worker-1    | 
myproj-worker-1    | Error: Invalid value for '-A' / '--app': 
myproj-worker-1    | Unable to load celery application.
myproj-worker-1    | While trying to load the module myproj.config the following error occurred:
myproj-worker-1    | Traceback (most recent call last):
myproj-worker-1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/bin/celery.py", line 57, in convert
myproj-worker-1    |     return find_app(value)
myproj-worker-1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/app/utils.py", line 384, in find_app
myproj-worker-1    |     sym = symbol_by_name(app, imp=imp)
myproj-worker-1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
myproj-worker-1    |     module = imp(module_name, package=package, **kwargs)
myproj-worker-1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/utils/imports.py", line 105, in import_from_cwd
myproj-worker-1    |     return imp(module, package=package)
myproj-worker-1    |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
myproj-worker-1    |     return _bootstrap._gcd_import(name[level:], package, level)
myproj-worker-1    |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
myproj-worker-1    |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
myproj-worker-1    |   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
myproj-worker-1    | ModuleNotFoundError: No module named 'myproj'
myproj-worker-1    | 
myproj-worker-1 exited with code 2

CodePudding user response：

You need to make sure that your project files are in the working directory when calling celery. In the case of your Dockerfile, this usually means setting your WORKDIR to the same location where you copy your project files. For example:

#...
WORKDIR /app
COPY . /app
# ...
CMD ["celery", "-A", "myproj.config", "-l", "info"]

Another way to ensure this always works correctly would be to create a setup.py for your django project and install it using pip (instead of copying the project files). That way, your app and worker should be able to be invoked irrespective of the working directory because it'll be importable from site-packages.

Example of this using a multi-stage docker build:

FROM python:3.9-slim as build
WORKDIR /build
RUN pip install wheel
COPY requirements.txt .
# create wheels for your dependencies
RUN python -m pip wheel -w /build/dist -r requirements.txt

# build your project to a wheel too
COPY . .
RUN pip wheel --no-deps --find-links /build/dist -w /build/dist  .

FROM python:3.9-slim as app
COPY --from=build /build/dist /wheelhouse
# install your project (using the name in setup.py)
WORKDIR /app
RUN pip install --no-index --find-links /wheelhouse my-project
# you only need to copy manage.py here since your project is in site-packages
COPY manage.py .
# ...