Elasticsearch - all replica shards are unassigned after configuring cluster awareness-CodePudding

I have an (Elasticsearch) 7 node cluster: 6 nodes are data and master eligible nodes. One is a witness and voting only node.

I have two datacenters, A1 and A2.

I have recently re-configured 6 of the data nodes with cluster routing allocation awareness, depending on the datacenter that they reside on:

node.attr.ABC: A1
cluster.routing.allocation.awareness.attributes: ABC

node.attr.ABC: A2
cluster.routing.allocation.awareness.attributes: ABC

Since I have set this and restarted all nodes, I see that all of my replica shards are now unassigned, and ONLY primary shards are assigned to nodes.

If I attempt to use allocate_replica and move one of the replicas to either node on either A1/A2 -

{
    "commands": [
        {
            "allocate_replica": {
                "index": "MY_INDEX",
                "shard": 0,
                "node": "node1.local"
            }
        }
    ]
}

I will get the error:

node does not contain the awareness attribute [A1]; required attributes cluster setting

How can I fix this and have my replica shards re-assigned correctly to the nodes?

CodePudding user response：

The response from GET _cluster/allocation/explain provides the following clue:

node does not contain the awareness attribute [A1]; required attributes cluster setting [cluster.routing.allocation.awareness.attributes=A1

It seems that your cluster settings contains

cluster.routing.allocation.awareness.attributes: A1

Instead of what you mentioned

cluster.routing.allocation.awareness.attributes: ABC