Home > Enterprise >  Elasticsearch and Kibana not working with Letsencrypt certs
Elasticsearch and Kibana not working with Letsencrypt certs

Time：09-19

Kibana can't connect to Elasticsearch using Letsencrypt signed certs

I am trying to run a 3 node elasticsearch cluster with kibana using letsencrypt certs. I have copy pasted the standard docker-compose.yml and environment file from the official elasticsearch documentation here, and successfully got it working with self signed certs without error. When i try and swap out the self signed certs for lets encrypt signed certs the elasticsearch cluster works but kibana stops working with the error

kibana_1  | [2022-09-12T18:52:55.669 00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate

The letsencrypt signed certs are all properly mounted into the container with the same owner/group and permissions as the self signed certs, for example

-rw-r----- 1 root     root   991 Sep 12 14:03 bundle.zip
drwxr-x--- 2 root     root  4096 Sep 12 13:38 ca
-rw-r----- 1 root     root  2512 Sep 12 13:38 ca.zip
-rw-r----- 1 root     root  1899 Sep 12 14:03 cert.pem
-rw-r----- 1 root     root  7610 Sep 12 13:38 certs.zip
-rw-r----- 1 root     root  3749 Sep 12 14:03 chain.pem
drwxr-x--- 2 root     root  4096 Sep 12 13:38 es01
drwxr-x--- 2 root     root  4096 Sep 12 13:38 es02
drwxr-x--- 2 root     root  4096 Sep 12 13:38 es03
-rw-r----- 1 root     root  5648 Sep 12 14:03 fullchain.pem
-rw-r----- 1 root     root   272 Sep 12 13:38 instances.yml
-rw-r----- 1 root     root  1826 Sep 12 14:03 intermediary.pem
-rw-r----- 1 root     root  1704 Sep 12 14:03 privkey.pem
-rw-r----- 1 root     root  1923 Sep 12 14:03 root.pem

Which i achieved by running these commands as suggested in the official docs

sudo find certs/ -type f -exec chmod 640 "{}" \;
sudo find certs -type d -exec chmod 750 "{}" \;

The intermediary.pem and root.pem certs were split out from the fullchain.pem cert and tried as part of the CA cert bundle as suggested in this other SO question. I have tried many different combinations of these certs in both the elastic search and kibana config and although more than one way works for the elastic search nodes, none of them work with kibana.

This is the final attempt i made, using privkey.pem as the key, fullchain.pem as the cert, and chain.pem as the CA, as suggested here in the elasticsearch docs. The below file omits the "setup" container that you will see in the official docs.

docker-compose.yml

version: "2.2"

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata01:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata02:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata03:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - kibanadata:/usr/share/kibana/data
      - ./certs:/usr/share/kibana/config/certs
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVER_HOST=0.0.0.0
      - SERVERNAME=dev.mysite.com
      - ELASTICSEARCH_HOSTS=https://dev.mysite.com:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/chain.pem
 ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://dev.mysite.com:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local

.env

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=asdf1234

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=asdf1234

# Version of Elastic products
STACK_VERSION=8.4.1

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

I have verified that elasticsearch is working both by the lack of errors in the output of the containers and also by running

curl -u elastic:asdf1234 https://dev.mysite.com:9200/_cluster/health

which gives me the output

{"cluster_name":"docker-cluster","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":11,"active_shards":22,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

CodePudding user response：

Unlike Elasticsearch for some reason in Kibana you cannot use any of the certs that letsencrypt gives you as the CA. You must find the public root CA cert of letsencrypt itself, which is isrgrootx1.pem and can be downloaded from letsencrypt.org/certs/isrgrootx1.pem. Unfortunately none of this is clear in any documentation anywhere and after days of fruitless searching I stumbled upon this in another SO question!

Once you have that cert you can bind it into the container and then update the config to look like this

version: "2.2"

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata01:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata02:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - esdata03:/usr/share/elasticsearch/data
      - ./certs:/usr/share/elasticsearch/config/certs
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - kibanadata:/usr/share/kibana/data
      - ./certs:/usr/share/kibana/config/certs
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVER_HOST=0.0.0.0
      - SERVERNAME=dev.mysite.com
      - ELASTICSEARCH_HOSTS=https://dev.mysite.com:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/isrgrootx1.pem
      - SERVER_SSL_ENABLED="true"
      - SERVER_SSL_KEY=/usr/share/kibana/config/certs/privkey.pem
      - SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/fullchain.pem
      - SERVER_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/chain.pem
 ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://dev.mysite.com:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local

Now everything should work fine!

Page link：https//www.codepudding.com/Enterprise/549579.html

Prev:Elasticsearch mirroring primary shards and allocating replicas only
Next:How do I call Future() only once when rebuild in Flutter?
  • Related

About Us:  Contact Us      Terms of Service       Privacy Policy

Copyright © 2010-2023，Powered By CodePudding