Am trying to implement an index template with datastream enabled and then set contains in ingest node pipelines. So that I could get metrics with below-mentioned index format :

.ds-metrics-kubernetesnamespace

I had tried this sometime back and I did these things as mentioned above and it was giving metrics in such format but now when I implement the same it's not changing anything in my index. I cannot see any logs in openshift cluster so ingest seems to be working fine(when I add a doc and test it works fine)

PUT _ingest/pipeline/metrics-index
{
   "processors": [
    {
      "set": {
      "field": "_index",
      "value": "metrics-{{kubernetes.namespace}}",
      "if": "ctx.kubernetes?.namespace==\"dev\""
       }
    }
   ]
}

This is the ingest node condition I have used for indexing.

    metricbeatConfig:
      metricbeat.yml: |
        metricbeat.modules:
        - module: kubernetes
          enabled: true
          metricsets:
            - state_node
            - state_daemonset
            - state_deployment
            - state_replicaset
            - state_statefulset
            - state_pod
            - state_container
            - state_job
            - state_cronjob
            - state_resourcequota
            - state_service
            - state_persistentvolume
            - state_persistentvolumeclaim
            - state_storageclass
            - event

CodePudding user response：

Since you're using Metricbeat, you have another way to do this which is much better.

Simply configure your elasticsearch output like this:

output.elasticsearch:
  hosts: ["http://<host>:<port>"]
  indices:
    - index: "%{[kubernetes.namespace]}"
      mappings:
        dev: "metrics-dev"
      default: "metrics-default"

or like this:

output.elasticsearch:
  hosts: ["http://<host>:<port>"]
  indices:
    - index: "metrics-%{[kubernetes.namespace]}"
      when.equals:
        kubernetes.namespace: "dev"
      default: "metrics-default"

or simply like this would also work if you have plenty of different namespaces and you don't want to manage different mappings:

output.elasticsearch:
  hosts: ["http://<host>:<port>"]
  index: "metrics-%{[kubernetes.namespace]}"

CodePudding user response：

Steps to create datastreams in elastic stack:

1. create an ILM policy
2. Create an index template that has an index pattern that matches with the index pattern of metrics/logs.(Set number of primary shards/replica shards and mapping in index template)
3. Set a condition in ingest pipeline.(Make sure no such index exist)

If these conditions meet it will create a data stream and logs/metrics would have an index starting with .ds- and it will be hidden in index management.

In my case the issue was I did not have enough permission to create a custom index. When I checked my OpenShift logs I could find metricbeat was complaining about the privilege. So I gave Superuser permission and then used ingest node to set conditional indexing

PUT _ingest/pipeline/metrics-index
{
   "processors": [
    {
      "set": {
      "field": "_index",
      "value": "metrics-{{kubernetes.namespace}}",
      "if": "ctx.kubernetes?.namespace==\"dev\""
       }
    }
   ]
}