Python script in Docker can't find module in subdirectory-CodePudding

My project structure is as follows:

.
├── my_script.Dockerfile
├── README.rst
├── log
│   └── my_script.log
├── pickles
├── requirements.txt
├── requirements_my_script.txt
├── src
│   ├── __init__.py
│   ├── __pycache__
│   ├── main.py
│   ├── modules
│   │   ├── __init__.py
│   │   └── module.py
│   ├── other_scripts
│   │   └── my_script.py
│   └── utils
│       ├── __init__.py
│       ├── __pycache__
│       └── logging_utils.py

my_script.Dockerfile is as follows:

# syntax=docker/dockerfile:1

FROM python:3.10-slim-buster
WORKDIR /src
COPY requirements_my_script.txt requirements_my_script.txt
RUN pip3 install -r requirements_my_script.txt
COPY src/. .
CMD ["python3", "other_scripts/my_script.py"]

my_script.py:

import logging

from src.utils.logging_utils import start_logger


def main(logger):
    # some code here
    logger.info('done')
    sleep(120)
       
    
if __name__=='__main__':
    logger = start_logger('my_script.log')
    try:
        logger.info('Starting..')
        main(logger)
    except Exception as e:
        logger.critical('Crashed with error: {}'.format(e))

And the logging_utils.py:

import os
import logging

from logging.handlers import RotatingFileHandler

def start_logger(filename, level='INFO'): # INFO or DEBUG
    # # LOGGING
    log_folder = os.path.join(os.path.expanduser('~'), 'project_folder_name', 'log')
    if not os.path.exists(log_folder):
            os.makedirs(log_folder)
    log_filename = filename

    logger = logging.getLogger()
    logFormatter = logging.Formatter(fmt='%(asctime)s :: %(levelname)s - %(message)s')
    if level == 'DEBUG':
        logger.setLevel(logging.DEBUG)
    else:
        logger.setLevel(logging.INFO)

    # add a rotating handler
    handler = RotatingFileHandler(os.path.join(log_folder, log_filename), maxBytes=1000000, backupCount=1)
    handler.setFormatter(logFormatter)
    logger.addHandler(handler)
    return logger

This runs flawlessly on my machine, but when I build the docker and try to run it I get the following error:

from src.utils.logging_utils import start_logger
ModuleNotFoundError: No module named 'src'

I tried several different COPY statements in the dockerfile but without success.

Could someone please tell me what I'm doing wrong?

EDIT I tried this dockerfile as well:

# syntax=docker/dockerfile:1

FROM python:3.10-slim-buster
WORKDIR /app
COPY requirements_my_script.txt requirements_my_script.txt
RUN pip3 install -r requirements_my_script.txt
COPY src/ src/
CMD ["python3", "src/other_scripts/my_script.py"]

But I get the exact same error.

CodePudding user response：

When your code does its import

from src.utils.logging_utils ...

Python looks for a ./src/utils/logging_utils.py, relative to the current directory and anything else in the Python search path. But in your Dockerfile, you strip out the src directory:

COPY src/. .

copies the contents of the src directory into the current directory. I'm guessing if you docker run --rm your-image ls, you'll see the utils directory immediately in the application directory and not in the src subdirectory.

If you need to copy a subdirectory from your host into a subdirectory in your image, you need to repeat the directory name on both sides of COPY, like

COPY src/ src/

CodePudding user response：

Remove this line from my_script.py:

from src.utils.logging_utils import start_logger

Add these lines to your my_script.py:

from sys import path as syspath
from os import path as ospath
syspath.append(ospath.join(ospath.dirname(ospath.realpath(__file__)),'..','utils')) # To find out where to look, import the utils path into the script path
from logging_utils import start_logger

And do this in Dockerfile :

COPY src/ src/