Remove ECS data from metricbeat for smaller documents-CodePudding

I use the graphite beat to get graphite protocol metrics into es. The metric document is much bigger than the metric data itself (timestamp, value, metric name).

I also get all the ECS data inserted and I think it will make my queries much slower (and my documents much bigger) and I don't need this data.

Can I remove the ECS data somehow in the metricbeat configuration?

CodePudding user response：

You might be able to use Metricbeat's drop_fields processor, but it might not be able to remove all the fields you specify as some are added after the processor chain.

So, acting on the ES side will guarantee you that you can change the event source the way you like. Also if you have many Beats deployed, you only need to configure this in a single place.

One way to achieve this is to create an index template for Metricbeat events and attach an ingest pipeline to it.

PUT _index_template/my-template
{
    "index_patterns" : [
      "metricbeat-*"
    ],
    "template" : {
      "settings" : {
        "index" : {
          "lifecycle" : {
            "name" : "metric-lifecycle"
          },
          "codec" : "best_compression",
          "default_pipeline" : "metric-pipeline"
        }
      },
   ...

Then the metric-pipeline would simply look like this and remove all the fields listed in the field array:

PUT _ingest/pipeline/metric-pipeline
{
  "processors": [
    {
      "remove": {
        "field": ["agent", "host", "..."]
      }
    }
  ]
}