Error connecting Java Spring application to ScyllaDB Cluster in Docker-CodePudding

I have ScyllaDB cluster running successfully on docker, but I'm having some connectivity problems when trying to connect my Spring Boot application to it.

The cluster seems to be healthy:

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.19.0.3  205.31 KB  256          ?       c387bd38-cf37-4c70-a8a3-db586f6410c4  Rack1
UN  172.19.0.2  101.61 KB  256          ?       89dda41e-70c5-43a5-a4a7-ce126007a1e1  Rack1
UN  172.19.0.4  210.34 KB  256          ?       58dfe868-a7ac-47a5-b3e9-dae610ef0bc9  Rack1

CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                            NAMES
b90d0357abee   scylladb/scylla:4.5.0   "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes   22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   scylla-node2
19f19cb2e631   scylladb/scylla:4.5.0   "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes   22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   scylla-node3
c98b0b1aa018   scylladb/scylla:4.5.0   "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes   22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   scylla-node1

And CQL works without any issues:

docker exec -it scylla-node1 cqlsh
Connected to  at 172.19.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh>

I'm trying to connect my Java app to the cluster using the datastax drivers (scylla-driver-core version 3.11.2.0):

Cluster cluster = Cluster.builder().addContactPoints("172.19.0.3", "172.19.0.2", "172.19.0.4")
                .build();
System.out.println("Connected to cluster "   cluster.getMetadata().getClusterName());

but I'm getting the following exception:

2022-05-08 18:00:02.928  INFO 4760 --- [  restartedMain] com.datastax.driver.core.NettyUtil       : Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2022-05-08 18:00:09.149  WARN 4760 --- [r1-nio-worker-0] com.datastax.driver.core.Connection      : Error creating netty channel to /172.19.0.4:9042

io.netty.channel.ConnectTimeoutException: connection timed out: /172.19.0.4:9042
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[netty-transport-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]

Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.19.0.4:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.4:9042] Cannot connect), /172.19.0.3:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.3:9042] Cannot connect), /172.19.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.2:9042] Cannot connect))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:270)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
    at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1921)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1832)
    at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:476)
    at com.dt.package.ScyllaConnection.connect_and_query(ScyllaConnection.java:15)
    at com.dt.package.MyApp.main(MyApp.java:12)
    ... 5 more

Example of docker-compose file:

  scylla-node2:
    container_name: scylla-node2
    image: scylladb/scylla:4.5.0
    restart: always
    command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
    volumes:
      - "./scylla/scylla.yaml:/etc/scylla/scylla.yaml"
      - "./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties"
    networks:
      web:

This same code works when trying to connect to a Scylla Cloud cluster (with the proper IPs). Am I missing something simple here?

CodePudding user response：

The problem can be motivated because you are trying to connect to the ScyllaDB nodes with their docker internal IP addresses.

If you need to reach your node from outside of the internal docker network you need to properly expose the appropriate ports to the docker host.

Using docker compose you can export a container port using the ports field. For example:

  scylla-node2:
    container_name: scylla-node2
    image: scylladb/scylla:4.5.0
    restart: always
    command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
    ports:
      - 9043:9042
    volumes:
      - "./scylla/scylla.yaml:/etc/scylla/scylla.yaml"
      - "./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties"
    networks:
      web:

In the example, we are exposing the container port 9042 to the host port 9043.

You will need similar configurations to the rest of the nodes: you are using the same host for all of them and as a consequence you need to pick a different port for every node.

  scylla-node1:
    ...
    ports:
      - 9042:9042
    ...
  scylla-node2:
    ...
    ports:
      - 9043:9042
    ...
  scylla-node3:
    ...
    ports:
      - 9044:9042
    ...

The different nodes will share the same host, the one in which the docker containers are running. To connect to them from your Spring application you need something like the following:

Cluster cluster =
  Cluster.builder()
    .addContactPointsWithPorts(
      new InetSocketAddress("127.0.0.1", 9042),
      new InetSocketAddress("127.0.0.1", 9043),
      new InetSocketAddress("127.0.0.1", 9044)
    );

System.out.println("Connected to cluster "   cluster.getMetadata().getClusterName());

Please, note the use of the method addContactPointsWithPorts instead of addContactPoints in the code.