I have installed Grafana, Loki, Promtail and Prometheus with the grafana/loki-stack
.
I also have Nginx set up with the Nginx helm chart.
Promtail is ingesting logs fine into Loki, but I want to customise the way my logs look. Specifically I want to remove a part of the log because it creates errors when trying to parse it with either logfmt
or json
(Error: LogfmtParserErr
and Error: JsonParserErr
respectively).
The logs look like this:
2022-02-21T13:41:53.155640208Z stdout F timestamp=2022-02-21T13:41:53 00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63
and I want to remove the part where it says stdout F
so the log will look like this:
2022-02-21T13:41:53.155640208Z timestamp=2022-02-21T13:41:53 00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63
I have figured out that on the ingestion side it could be something with Promtail, but ist it also possible to make a LogQL query in Loki to just replace that string? And how would one set up the Promtail configuration for the wanted behaviour?
CodePudding user response:
Promtail should be configured to replace the string with the replace
stage.
Here is a sample config that removes the stdout F
part of the log for all logs coming from the namespace ingress.
promtail:
enabled: true
pipelineStages:
- docker: {}
- match:
selector: '{namespace="ingress"}'
stages:
- replace:
expression: "(stdout F)"
replace: ""
Specifically this example works for the grafana/loki-stack
chart.