GKE cluster node ends up with CrashLoopBackOff-CodePudding

I've had a 3 node setup in GKE. And one of my pod creation is in CrashLoopBackOff state and it is not recovering. The log suggests the below java.lang.IllegalArgumentException. But the other 2 pods they have no such issue. They are up and running. I'm completely unsure of the issue, can someone help me?

Is the issue, a by-product of install-plugins in the YML file? If yes, why didn't the same problem occur with other pods? Can you please help me with it?

Exception:

    "type": "server", "timestamp": "2022-08-29T19:52:29,743Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "dev", "node.name": "dev-es-data-hot-1", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]",
"Caused by: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561) ~[elasticsearch-7.16.3.jar:7.16.3]",
uncaught exception in thread [main]
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:137) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:500) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]",
"... 6 more"] }

Here is my YAML config:

 - name: data-hot-ingest
   count: 3
   config:
     node.roles: ["data_hot", "ingest", "data_content"]
     node.attr.data: hot
     node.store.allow_mmap: false
     xpack.security.authc:
       anonymous:
         username: anon
         roles: monitoring_user
   podTemplate:
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: type
                 operator: In
                 values:
                 - hot
       initContainers:
       - name: install-plugins
         command:
         - sh
         - -c
         - |
           bin/elasticsearch-plugin install --batch repository-gcs
       - name: set-virtual-mem
         command:
         - sysctl
         - -w
         - vm.max_map_count=262144
       containers:
         - name: elasticsearch
           resources:
             requests:
               memory: "64Gi"
               cpu: "30000m"
             limits:
              memory: "65Gi"
              cpu: "30000m"
           env:
             - name: ES_JAVA_OPTS
               value: -Xms32g -Xmx32g
           readinessProbe:
             httpGet:
               scheme: HTTPS
               port: 8080
   volumeClaimTemplates:
   - metadata:
       name: elasticsearch-data
     spec:
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 350Gi
       storageClassName: gold

EDIT: We have this secure setting configured, which is linked to a secret in our

  secureSettings:
  - secretName: credentials

CodePudding user response：

[ANSWERING MY OWN QUESTION]

Trying to resolve the below exception: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json]

I tried comparing the yaml config of the pods, and I found that the pods running successfully do not have a secure setting. But the pod that was crash looping, had the secure setting under elastic-internal-secure-settings

- name: elastic-internal-secure-settings
    secret:
      defaultMode: 420
      optional: false
      secretName: dev-es-secure-settings

And in the operator yaml, I found this:

secureSettings:
  - secretName: credentials

Just to confirm the behaviour, I upscaled the statefulset, and found the new pod also crash looping with the same error. So someone had tried the secure setting last month, and it crash looped the pod, and didn't reset it back to normal. Once I removed the secure-setting from the operator yaml, the pods started running without any issue.