Getting error while installing elasticsearch Statefulset-CodePudding

I am trying to install an elastic statefulset in my GKE cluster but it's throwing an error and am unable to identify the error here this is the log that I had got inside the pod. Can someone help me? I have given the error logs as well as the elasticsearch_statefulset.yml file.

{"type": "server", "timestamp": "2021-12-16T09:30:49,473Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "k8s-logs", "node.name": "es-cluster-0", "message": "failed to resolve host [es-cluster-2.elasticsearch]",
"stacktrace": ["java.net.UnknownHostException: es-cluster-2.elasticsearch",
"at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1495) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1354) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1288) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:548) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:490) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:855) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

This is the yml I used to configure the stateful set :

apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: es-cluster
    namespace: kube-logging
  spec:
    serviceName: elasticsearch
    replicas: 3
    selector:
      matchLabels:
        app: elasticsearch
    template:
      metadata:
        labels:
          app: elasticsearch
      spec:
        containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
          resources: {}
          ports:
          - containerPort: 9200
            name: rest
            protocol: TCP
          - containerPort: 9300
            name: inter-node
            protocol: TCP
          volumeMounts:
          - name: data
            mountPath: /usr/share/elasticsearch/data
          env:
            - name: cluster.name
              value: k8s-logs
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: discovery.seed_hosts
              value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
            - name: cluster.initial_master_nodes
              value: "es-cluster-0,es-cluster-1,es-cluster-2"
            - name: ES_JAVA_OPTS
              value: "-Xms512m -Xmx512m"
        initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
          securityContext:
            privileged: true
          volumeMounts:
          - name: data
            mountPath: /usr/share/elasticsearch/data
        - name: increase-vm-max-map
          image: busybox
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
        - name: increase-fd-ulimit
          image: busybox
          command: ["sh", "-c", "ulimit -n 65536"]
          securityContext:
            privileged: true
    volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: elasticsearch
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: standard
        resources:
          requests:
            storage: 10Gi

Service file which I used :

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  type: LoadBalancer
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

Output for kubectl get statefulset -n kube-logging es-cluster -oyaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-12-20T06:31:55Z"
  generation: 1
  name: es-cluster
  namespace: kube-logging
  resourceVersion: "43285"
  uid: a1730c94-1aa5-461c-ba73-9af617ea4c42
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: elasticsearch-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: es-cluster-0.elasticsearch
        - name: cluster.initial_master_nodes
          value: es-cluster-0
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        image: busybox
        imagePullPolicy: Always
        name: fix-permissions
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - command:
        - sh
        - -c
        - ulimit -n 65536
        image: busybox
        imagePullPolicy: Always
        name: increase-fd-ulimit
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: gold
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: es-cluster-6f5bd744c7
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updateRevision: es-cluster-6f5bd744c7
  updatedReplicas: 1

Output for kubectl get svc -n kube-logging elasticsearch-headless -oyaml :

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
  creationTimestamp: "2021-12-20T06:31:40Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: elasticsearch
  name: elasticsearch-headless
  namespace: kube-logging
  resourceVersion: "43384"
  uid: 80d10b9b-1f22-48ce-ba39-7d5801dc91ee
spec:
  clusterIP: 10.8.4.128
  clusterIPs:
  - 10.8.4.128
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: rest
    nodePort: 30625
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: inter-node
    nodePort: 30056
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 35.222.11.96

CodePudding user response：

Additionally to the service you created to expose elastic search outside the cluster, you also need an headless service so that each node / Pod of the elastic cluster can communicate with each other.

I would do the following:

First, inside the spec of the StatefulSet, change spec.serviceName to another value, such as for example elasticsearch-headless

Second. create the new service with the following:

apiVersion: v1
kind: Service
metadata:
  # must be the same as the StatefulSet spec.serviceName
  name: elasticsearch-headless
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  type: ClusterIP
  # headless service, can be used by elastic Pods to contact each other
  clusterIP: None
  ports:
  - name: rest
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: inter-node
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch

Some docs on Headless Services

Also, you may be interested into checking HELM Charts and ECK because there is several stuff ready to be used in order to deploy production-ready elastic search clusters.