Converting standard string based logging to json key-value based in Python-CodePudding

my requirement is to create json(dictionary key value based) python logging from standard string based logging as we need to push all logs to Splunk and key/value based logging can be better searched,queries in logs aggregators tools.

I am using below code for that.It is generating key/value based logging but it is also printing standard(string based logs).

Please suggest how to get only structured(key/value) based logs not the standard string based in Python

Thanks

CodePudding user response：

Just change logger name from "json" to "root", this will replace default string based logger:

import logging
import json

def logging_override(name: str, extra: dict):
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)
    stream_handler = logging.StreamHandler()
    basic_dict = {"time": "%(asctime)s", "level": "%(levelname)s", "message": "%(message)s"}
    full_dict = {**basic_dict, **extra}
    stream_formatter = logging.Formatter(json.dumps(full_dict))
    stream_handler.setFormatter(stream_formatter)
    if not logger.handlers:
        logger.addHandler(stream_handler)
    logger.handlers[0] = stream_handler
    logger = logging.LoggerAdapter(logger, extra)
    return logger

def main():
    extra = {"server_ip": "192.168.1.2", "username": "Jack Sparrow"}
    logger = logging_override("root", extra)
    logger.info("Sending Email to username: 'Jack Sparrow' regarding server_ip: '192.168.1.2'")

if __name__ == '__main__':
    main()

# Output:
# {"time": "2021-11-18 10:27:37,253", "level": "INFO", "message": "Sending Email to username: 'Jack Sparrow' regarding server_ip: '192.168.1.2'", "server_ip": "192.168.1.2", "username": "Jack Sparrow"}

CodePudding user response：

This is supported using the following cookbook recipe:

import json
import logging

class StructuredMessage:
    def __init__(self, message, /, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        return '%s >>> %s' % (self.message, json.dumps(self.kwargs))

_ = StructuredMessage   # optional, to improve readability

logging.basicConfig(level=logging.INFO, format='%(message)s')
logging.info(_('message 1', foo='bar', bar='baz', num=123, fnum=123.456))

When run, this prints

message 1 >>> {"fnum": 123.456, "num": 123, "bar": "baz", "foo": "bar"}

There's some more options described in the linked recipe.