How to pass broker_url from Django settings.py to a Celery service-CodePudding

I have Celery running as a service on Ubuntu 20.04 with RabbitMQ as a broker.

Celery repeatedly restarts because it cannot access the RabbitMQ url (RABBITMQ_BROKER), a variable held in a settings.py outside of the Django root directory.

The same happens if I try to initiate celery via command line.

I have confirmed that the variable is accessible from within Django from a views.py print statement.

If I place the RABBITMQ_BROKER variable inside the settings.py within the Django root celery works.

My question is, how do I get celery to recognise the variable RABBITMQ_BROKER when it is placed in /etc/opt/mydjangoproject/settings.py?

My celery.py file:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mydjangoproject.settings')

app = Celery('mydjangoproject')

default_config = 'mydjangoproject.celery_config'

app.config_from_object(default_config)
app.autodiscover_tasks()

My celery_config.py file:

from django.conf import settings

broker_url = settings.RABBITMQ_BROKER
etc...

The settings.py in /etc/opt/mydjangoproject/ (non relevant stuff deleted):

from mydangoproject.settings import *

RABBITMQ_BROKER = 'amqp://rabbitadmin:somepassword@somepassword@webserver:5672/mydangoproject'
etc...

My /etc/systemd/system/celery.service file:

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=DJANGO_USER
Group=DJANGO_USER
EnvironmentFile=/etc/conf.d/celery
WorkingDirectory=/opt/mydjangoproject

ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
   --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
   --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE} --loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
    --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always

[Install]
WantedBy=multi-user.target

My /etc/conf.d/celery file:

CELERYD_NODES="worker"
CELERY_BIN="/opt/mydjangoproject/venv/bin/celery"
CELERY_APP="mydjangoproject"
CELERYD_CHDIR="/opt/mydjangoproject/"
CELERYD_MULTI="multi"
CELERYD_OPTS="--time-limit=300 --without-heartbeat --without-gossip --without-mingle"
CELERYD_PID_FILE="/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

CodePudding user response：

Add the following line to the end of /etc/opt/mydjangoproject/settings.py to have celery pick up the correct broker url (casing might vary based on the version of celery you are using):

BROKER_URL = broker_url = RABBITMQ_BROKER

This will put the configuration in a place where it will be read by the call to celery's config_from_object function.

Next, you will also have to add an environment variable to your systemd unit. Since you are accessing settings as mydjangoproject.settings, you have to make the parent of the mydjangoproject directory accessible in the PYTHONPATH:

Environment=PYTHONPATH=/etc/opt

The PYTHONPATH provides python a list of directories to try when trying the import. However, because we have two different directories with the same name that we are using as a single package, we also have to add the following lines to /etc/opt/mydjangoproject/__init__.py and /opt/mydjangoproject/__init__.py:

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

CodePudding user response：

I solved this by adding the following to /etc/systemd/system/celery.service

Environment="PYTHONPATH=/etc/opt/mydjangoproject:/opt/mydjangoproject"
Environment="DJANGO_SETTINGS_MODULE=settings"