Have setup a 3-node Elasticsearch cluster using docker-compose. Followed below steps: On one of the master nodes, es11, gets below error, however same curl command works fine on other 2 nodes i.e. es12, es13:

Error:

curl -X GET 'https://localhost:9316'
    curl: (35) Encountered end of file

Below error in logs:

"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es13][SOMEIP:9316][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.transport.ConnectTransportException: [es11][SOMEIP:9316] handshake failed. unexpected remote node {es13}{SOMEVALUE}{SOMEVALUE
"at org.elasticsearch.transport.TransportService.lambda$connectionValidator$6(TransportService.java:468) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:95) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.transport.TransportService.lambda$handshake$9(TransportService.java:577) ~[elasticsearch-7.17.6.jar:7.17.6]",

https://localhost:9316 on browser gives site can't be reached error as well.It seems SSL certificate as created in step 4 below is having some issues in es11. Any leads please? OR If I repeat step 4, do i need to copy the certs again to es12 & es13?

Below elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

Ports as defined in all 3 nodes docker-compose.yml

 environment:
      - node.name=es11
      - transport.port=9316
 ports:
      - 9216:9200
      - 9316:9316

1. Initialize a docker swarm. On ES11 run docker swarm init. Follow the instructions to join 12 and 13 to the swarm.
2. Create an overlay network docker network create -d overlay --attachable elastic
3. If necessary, bring down the current cluster and remove all the associated volumes by running docker-compose down -v
4. Create SSL certificates for ES with docker-compose -f create-certs.yml run --rm create_certs
5. Copy the certs for es12 and 13 to the respective servers
6. Use this busybox to create the overlay network on 12 and 13 sudo docker run -itd --name containerX --net [network name] busybox
7. Configure certs on 12 and 13 with docker-compose -f config-certs.yml run --rm config_certs
8. Start the cluster with docker-compose up -d on each server
9. Set the passwords for the built-in ES accounts by logging into the cluster docker exec -it es11 sh then running bin/elasticsearch-setup-passwords interactive --url localhost:9316

CodePudding user response：

(as per your https://discuss.elastic.co thread)

you cannot talk HTTP to the transport protocol port, which you have defined in transport.port. you need to talk to port 9200 in the container, which you have mapped to 9216 outside the container

the transport port runs a binary protocol that is not HTTP accessible