Customize key value in python structured (json) logging from config file-CodePudding

I have to output my python job's logs as structured (json) format for our downstream datadog agent to pick them up. Crucially, I have requirements about what specific log fields are named, e.g. there must be a timestamp field which cannot be called e.g. asctime. So a desired log looks like:

{"timestamp": "2022-11-10 00:28:58,557", "name": "__main__", "level": "INFO", "message": "my message"}

I can get something very close to that with the following code:

import logging.config
from pythonjsonlogger import jsonlogger


logging.config.fileConfig("logging_config.ini", disable_existing_loggers=False)
logger = logging.getLogger(__name__)

logger.info("my message")

referencing the following logging_config.ini file:

[loggers]
keys = root

[handlers]
keys = consoleHandler

[formatters]
keys=json

[logger_root]
level=DEBUG
handlers=consoleHandler

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=json

[formatter_json]
class = pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s %(name)s - %(levelname)s:%(message)s

...however, this doesn't allow flexibility about the keys in the outputted log json objects. e.g. my timestamp object is called "asctime" as below:

{"asctime": "2022-11-10 00:28:58,557", "name": "__main__", "levelname": "INFO", "message": "my message"}

I still want that asctime value (e.g. 2022-11-10 00:28:58,557), but need it to be referenced by a key called "timestamp" instead of "asctime". If at all possible I would strongly prefer a solution that adapts the logging.config.ini file (or potentially a yaml logging config file) with relatively minimal extra python code itself.

I also tried this alternative python json logging library which I thought provided very simple and elegant code, but unfortunately when I tried to use that, I didn't get my log statement to output at all...

CodePudding user response：

You'll need to have a small, minimal amount of Python code, something like

# in mymodule.py, say

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record['timestamp'] = datetime.datetime.fromtimestamp(record.created).strftime('%Y-%m-%d %H:%M:%S')   f',{int(record.msecs)}'

and then change the configuration to reference it:

[formatter_json]
class = mymodule.CustomJsonFormatter
format=%(timestamp)s %(name)s - %(levelname)s:%(message)s

which would then output e.g.

{"timestamp": "2022-11-10 11:37:25,153", "name": "root", "levelname": "DEBUG", "message": "foo"}