Kibana says "Kibana server is not ready yet." and "elasticsearch-reset-password"-CodePudding

I am new to Elasticsearch/Kibana and am trying to set up a basic installation via Docker. I've backed myself into a corner, and I need help finding my way out.

I have the following docker-compose.yml.

services:

  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:8.4.0
    environment:
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - IPC_LOCK
    ports:
      - "9200:9200"

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:8.4.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticssearch:9200
    ports:
      - "5601:5601"

I run docker compose up . and the logs look mostly good. However, when I try to connect to http://localhost:5601/, I see a message "Kibana server is not ready yet." that never goes away.

The end of the Elasticsearch log looks like this.

{"@timestamp":"2022-08-26T15:26:25.616Z", "log.level":"ERROR", "message":"exception during geoip databases update", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1de6b5b3d4cb][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"vGjmfQNWTRS2sEeG0AiwuQ","elasticsearch.node.id":"3CcC2gJmRk2tQZOQTwU9HA","elasticsearch.node.name":"1de6b5b3d4cb","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.ElasticsearchException","error.message":"not all primary shards of [.geoip_databases] index are active","error.stack_trace":"org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:134)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48)\n\tat [email protected]/org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
2022-08-26T15:26:26.005783998Z {"@timestamp":"2022-08-26T15:26:26.002Z", "log.level": "INFO",  "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0]]]).","previous.health":"RED","reason":"shards started [[.geoip_databases][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1de6b5b3d4cb][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"vGjmfQNWTRS2sEeG0AiwuQ","elasticsearch.node.id":"3CcC2gJmRk2tQZOQTwU9HA","elasticsearch.node.name":"1de6b5b3d4cb","elasticsearch.cluster.name":"docker-cluster"}
2022-08-26T15:26:26.264786433Z {"@timestamp":"2022-08-26T15:26:26.264Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1de6b5b3d4cb][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"vGjmfQNWTRS2sEeG0AiwuQ","elasticsearch.node.id":"3CcC2gJmRk2tQZOQTwU9HA","elasticsearch.node.name":"1de6b5b3d4cb","elasticsearch.cluster.name":"docker-cluster"}
2022-08-26T15:26:26.304814423Z {"@timestamp":"2022-08-26T15:26:26.304Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1de6b5b3d4cb][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"vGjmfQNWTRS2sEeG0AiwuQ","elasticsearch.node.id":"3CcC2gJmRk2tQZOQTwU9HA","elasticsearch.node.name":"1de6b5b3d4cb","elasticsearch.cluster.name":"docker-cluster"}
2022-08-26T15:26:27.017126446Z {"@timestamp":"2022-08-26T15:26:27.016Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1de6b5b3d4cb][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"vGjmfQNWTRS2sEeG0AiwuQ","elasticsearch.node.id":"3CcC2gJmRk2tQZOQTwU9HA","elasticsearch.node.name":"1de6b5b3d4cb","elasticsearch.cluster.name":"docker-cluster"}

I'm not sure if that ERROR about "geoip databases" is a problem. It does look like cluster health is "GREEN".

The end of the Kibana logs looks like this.

[2022-08-26T15:26:25.032 00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
2022-08-26T15:26:25.091816903Z [2022-08-26T15:26:25.091 00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
2022-08-26T15:26:26.081102019Z [2022-08-26T15:26:26.080 00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
2022-08-26T15:26:26.155818080Z [2022-08-26T15:26:26.155 00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND elasticssearch
2022-08-26T15:26:26.982333104Z [2022-08-26T15:26:26.981 00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell

That "Unable to retrieve version information from Elasticsearch nodes." ERROR looks more like it could be a problem, but I'm not sure what to do about it. One online question that sounds similar comes down to the difference between ELASTICSEARCH_HOSTS and ELASTICSEARCH_URL for an earlier version of Elastic that doesn't seem relevant here.

Poking around online also turns up situations in which the "Kibana server is not ready yet." error is a problem with the security setup. The whole security setup part is a bit confusing to me, but it seems like one thing that might have happened is that I failed to setup passwords correctly. I'm trying to start over, so I shelled into the Elasticsearch instance and ran elasticsearch-reset-password --username elastic. I saw the following error.

elasticsearch@1de6b5b3d4cb:~$ elasticsearch-reset-password --username elastic
15:24:34.593 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [172.18.0.2]; the server provided a certificate with subject name [CN=1de6b5b3d4cb], fingerprint [cc4a98abd8b44925c631d7e4b05f048317c8e02b], no keyUsage and extendedKeyUsage [serverAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [IP:172.18.0.3,DNS:localhost,IP:127.0.0.1,DNS:1de6b5b3d4cb]; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [ba8730cc6481e4847e4a14eff4f774ca1c96ad0b] {trusted issuer}) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: Composite-Trust{JDK-trusted-certs,StoreTrustConfig{path=certs/http.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX}})])
java.security.cert.CertificateException: No subject alternative names matching IP address 172.18.0.2 found

Those are all the problems I have encountered. I don't know what they mean or which are significant, and Googling doesn't turn up any clear next steps. Any suggestions as to what is going on here?

CodePudding user response：

Never mind. Stupid mistake. I misspelled elasticsearch in the line.

ELASTICSEARCH_HOSTS=http://elasticssearch:9200

"ss" instead of "s". Easy to overlook. The error message in the Kibana logs was telling me what the problem was. I just didn't know how to interpret it.

Even though this was just a typo I'm going to leave this question up in case someone makes the same mistake and gets confused in the same way.