Bootstrapped index is set as the write index but logs are getting written to old index-CodePudding

We're running Elastic Fluentbit Kibana stack on kubernetes for container logs and it was working correctly with daily rollover based on date(new-YYYY-MM-DD) but on high volume it caused over shard size issue so created ILM policy mentioned below so that it can rollover quickly. Bootstrapped index is writable but still the old index of (new-YYYY-MM-DD) is getting written instead of the new index new-YYYY-MM-DD-000001. I have mentioned the things tried but no luck yet.

Created new policy with following condition:

PUT /_ilm/policy/new_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age" : "10m" #just to test faster, actually want to set it to an hour.
          }
        }
      },
      "delete": {
        "min_age": "20d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Created template:

PUT _template/new_template
{
  "index_patterns": ["new*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "new_policy",
    "index.lifecycle.rollover_alias": "new-alias" 
  }
}

Bootstrapped new index which is creating and rolling over correctly.

PUT /
{
  "aliases": {
    "new-alias":{
      "is_write_index": true 
    }
  }
}

Output plugin section inside fluentbit-configmap

[OUTPUT]
    Name            es
    Match           *
    Host            ${FLUENT_ELASTICSEARCH_HOST}
    Port            ${FLUENT_ELASTICSEARCH_PORT}
    HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
    HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format On
    Logstash_Prefix new
    Trace_Error     On
    Replace_Dots    On
    Retry_Limit     False
    tls             On
    tls.verify      Off

Tried indexing but no luck.

POST _reindex
{
  "source": {
    "index": "new-2021.09.30" 
  },
  "dest": {
    "index": "new-2021.09.30-000001", 
    "op_type": "create" 
  }
}

Also, tried creating entire new index-pattern but it seems coming from any system defaults and not making my ILM index as default for new logs.

alias           index                       filter routing.index routing.search is_write_index
new-alias new-2021.09.30-000001 -      -             -              false
new-alias new-2021.09.30-000002 -      -             -              false
new-alias new-2021.09.30-000003 -      -             -              false
new-alias new-2021.09.30-000004 -      -             -              false
new-alias new-2021.09.30-000005 -      -             -              false
new-alias new-2021.09.30-000006 -      -             -              false
new-alias new-2021.09.30-000007 -      -             -              false
new-alias new-2021.09.30-000008 -      -             -              false
new-alias new-2021.09.30-000009 -      -             -              false
new-alias new-2021.09.30-000010 -      -             -              false
new-alias new-2021.09.30-000011 -      -             -              false
new-alias new-2021.09.30-000012 -      -             -              false
new-alias new-2021.09.30-000013 -      -             -              true

CodePudding user response：

In your Fluentbit configuration you need to change the following:

Logstash_Format Off
Index new-alias

And remove Logstash_Prefix new

That's it. Since ILM will take care of naming the indexes and rolling them over, you simply need to write to new-alias and ILM takes care of the rest. No need for Fluentbit to make up the index name anymore.