Home > Net >  Remove a part of a log in Loki
Remove a part of a log in Loki

Time:02-25

I have installed Grafana, Loki, Promtail and Prometheus with the grafana/loki-stack.

I also have Nginx set up with the Nginx helm chart.

Promtail is ingesting logs fine into Loki, but I want to customise the way my logs look. Specifically I want to remove a part of the log because it creates errors when trying to parse it with either logfmt or json (Error: LogfmtParserErr and Error: JsonParserErr respectively).

The logs look like this:

2022-02-21T13:41:53.155640208Z stdout F timestamp=2022-02-21T13:41:53 00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63

and I want to remove the part where it says stdout F so the log will look like this:

2022-02-21T13:41:53.155640208Z timestamp=2022-02-21T13:41:53 00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63

I have figured out that on the ingestion side it could be something with Promtail, but ist it also possible to make a LogQL query in Loki to just replace that string? And how would one set up the Promtail configuration for the wanted behaviour?

CodePudding user response:

Promtail should be configured to replace the string with the replace stage.

Here is a sample config that removes the stdout F part of the log for all logs coming from the namespace ingress.

promtail:
  enabled: true
  pipelineStages:
  - docker: {}
  - match:
      selector: '{namespace="ingress"}'
      stages:
      - replace:
          expression: "(stdout F)"
          replace: ""

Specifically this example works for the grafana/loki-stack chart.

  • Related