How to configure FastAPI to publish logs to CloudWatch?-CodePudding

I have a FastAPI service that works as expected in every regard except the logging, only when it runs as a AWS Lambda function.

When running it locally the logs are displayed on the console as expected:

INFO:     127.0.0.1:62160 - "POST /api/v1/feature-requests/febbbc21-9650-44e6-8df5-80c8bb33b6ea/upvote HTTP/1.1" 200 OK
INFO:     127.0.0.1:62158 - "OPTIONS /api/v1/feature-requests HTTP/1.1" 200 OK
INFO:     127.0.0.1:62160 - "GET /api/v1/feature-requests HTTP/1.1" 200 OK
INFO:     127.0.0.1:62158 - "OPTIONS /api/v1/feature-requests-meta HTTP/1.1" 200 OK
INFO:     127.0.0.1:62160 - "GET /api/v1/feature-requests-meta HTTP/1.1" 200 OK
INFO:     127.0.0.1:62160 - "GET /api/v1/feature-requests/febbbc21-9650-44e6-8df5-80c8bb33b6ea HTTP/1.1" 200 OK
INFO:     127.0.0.1:62160 - "GET /api/v1/feature-requests-meta/febbbc21-9650-44e6-8df5-80c8bb33b6ea HTTP/1.1" 200 OK

However, when deployed as a Lambda function the logs are not there:

2022-09-07T10:44:57.426 02:00   START RequestId: fd44ae47-5bfb-42e3-aeb4-d9f29857bb39 Version: $LATEST
2022-09-07T10:44:57.604 02:00   END RequestId: fd44ae47-5bfb-42e3-aeb4-d9f29857bb39
2022-09-07T10:44:57.604 02:00   REPORT RequestId: fd44ae47-5bfb-42e3-aeb4-d9f29857bb39 Duration: 177.85 ms Billed Duration: 178 ms Memory Size: 2048 MB Max Memory Used: 152 MB Init Duration: 1733.88 ms
2022-09-07T10:45:00.299 02:00   START RequestId: 08a7a6da-c2c6-446c-baa3-1d08c9816f5b Version: $LATEST
2022-09-07T10:45:00.318 02:00   END RequestId: 08a7a6da-c2c6-446c-baa3-1d08c9816f5b

Even for the logs that are produced by our code (as opposed to the framework) are not visible when running as a Lambda function.

Configuration:

In app.py

LOG = logging.getLogger()
log_format = "%(asctime)s %(levelname)s %(message)s"
log_date_fmt = "%Y-%m-%d %H:%M:%S"
logging.basicConfig(
    format=log_format,
    level=logging.INFO,
    datefmt=log_date_fmt,
)

In every other Python file:

LOG = logging.getLogger(__name__)

logging.conf

[loggers]
keys=root,api,config

[handlers]
keys=console_handler

[formatters]
keys=normal_formatter

[logger_root]
level=INFO
handlers=console_handler

[logger_api]
level=INFO
handlers=console_handler
qualname=api
propagate=0

[logger_config]
level=INFO
handlers=console_handler
qualname=config
propagate=0

[handler_console_handler]
class=StreamHandler
level=INFO
formatter=normal_formatter
args=(sys.stdout,)

[formatter_normal_formatter]
format=%(asctime)s %(levelname)s %(name)s %(message)s
datefmt=%Y-%m-%d %H:%M:%S

I am not sure what else needs to happen to get the logs in CloudWatch.

CodePudding user response：

try cloudwatch library

Run pip install cloudwatch in the console

then in your code:

import logging
from cloudwatch import cloudwatch

logger = logging.getLogger('cloudwatch_logger')
formatter = logging.Formatter('%(asctime)s : %(levelname)s - %(message)s')
handler = cloudwatch.CloudwatchHandler(log_group = 'cloudwatch_log_group')
handler.setFormatter(formatter)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

and then either use this logger to log to CloudWatch logger.warning('I am here') or add the handler to the root logger via logging configuration

CodePudding user response：

As it turns out the following setup is needed:

On the top of the logging.conf above uvicorn has to be imported that creates an extra property on logging and than fileConfig has to be used like this:

import uvicorn
logging.config.fileConfig("logging.conf", disable_existing_loggers=False)
LOG = logging.getLogger(__name__)