I've configured an Akka cluster (2 nodes) and can run it on free instances of Oracle Cloud. However, I have to keep all TCP ports open. According to netstat -ano -p tcp, other than ports I have in the seed nodes, two other ports are used and their value is not fixed.
If I only keep the two seed ports open, then cluster nodes cannot find each other. Any idea where these random ports are coming from?
(I am using Akka stream and HTTP, but I doubt that they are related to Akka cluster connecting to seed nodes).
EDIT: Below you can see my configurations and the main file. I do use Akka HTTP and Akka Stream and connect to WebSocket (through HTTP). But as I already said, I don't think these are related to Akka cluster nodes not being able to find each other.
akka {
log-dead-letters = off
http.client.websocket.periodic-keep-alive-max-idle = 2 seconds
actor {
serialize-messages = off
allow-java-serialization = off
warn-about-java-serializer-usage = on
provider = cluster
serializers {
proto = "akka.remote.serialization.ProtobufSerializer"
myown = "myown.Serializer"
}
serialization-bindings {
"com.google.protobuf.Message" = proto
"myown.Serializee" = myown
}
}
remote {
artery {
transport = tcp
canonical.hostname = ${akka.clustering.public.ip}
canonical.port = ${akka.clustering.port}
bind.hostname = ${akka.clustering.private.ip}
bind.port = ${akka.clustering.port} # Yes, I use the same port
}
}
cluster {
seed-nodes = [
"akka://[email protected]:33221",
"akka://[email protected]:22334"]
downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
}
clustering {
public.ip = 127.0.0.1
private.ip = 127.0.0.1
port = 10000
# Node 1:
# AKKA_PUBLIC_IP : 123.124.125.126
# AKKA_PRIVATE_IP: 10.0.0.100
# AKKA_PORT : 33221
# Node 2:
# AKKA_PUBLIC_IP : 234.123.125.126
# AKKA_PRIVATE_IP: 10.0.0.200
# AKKA_PORT : 22334
public.ip = ${?AKKA_PUBLIC_IP}
private.ip = ${?AKKA_PRIVATE_IP}
port = ${?AKKA_PORT}
}
}
@main
def main(): Unit =
def behavior = Behaviors.setup[Nothing] { ctx =>
// Actor1 and Actor2 are two simple actors
ctx.spawn(Actor1(/*config-object*/), "Actor1.id")
ctx.spawn(Actor2(/*config-object*/), "Actor2.id")
Behaviors.empty
}
// I do not manipulate configuration
ActorSystem[Nothing](behavior, "MySystem", ConfigFactory.load())
sudo ufw status gives me Status: inactive. I am using Ingress/Egress rules that can be defined for Virtual Cloud Networks on Oracle instances. If I restrict either one of the Source Port Range or Destination Port Range, nodes won't find each other.
CodePudding user response：
It's hard to know without seeing your config. But I'll take some guesses. If these don't work, post your config file and the code you use to initialize the actor system. (Because it would help to know if you are using bootstrap, for example.)
First of all, there really isn't a "seed port". There is a port that Akka remoting listens on (by default 25220, although it's often overridden). That remoting port is the only one that needs to be open in order for cluster formation to happen. (As I mention below, there are some management and monitoring ports you may wish to open, but those aren't needed for cluster formation.) "Seed nodes" are one way of doing discovery and basically is a list of known nodes (both host and port). The point being that there is no special port for seeding/discovery: it uses the same remoting port. The hosts/port you specify in the seed node list are the places you are connecting to, not the ones you are listening on.
The fact that the ports you are seeing are not fixed makes me suspect that you might be running some of the akka cluster samples. In order to demonstrate clustering on a single machine, these samples often start several JVMs that all listen on random ports. It does this so that the nodes don't conflict with each other. An example of this is here where a port of 0 is specified, which results in a random port being chosen. But that's just for demo purposes, and under normal production configs there shouldn't be any non-fixed ports listening.
Again, the entire Akka remoting system (upon which Clustering is an additional layer) runs on a single specified port. This value is specified under remote.artery.canonical.port (presuming you are using the default Artery networking). If you are seeing multiple ports getting opened, especially with non-fixed values, look at what you have specified in your config for that value, as well as potentially places you are overriding that in your code. (Such as here: where the sample code I mentioned overrides the random port with values specified on the command line.)
However, even though I say that Akka remoting runs on a single port, there are a couple of other subsystems that could be opening ports. One is cluster HTTP management. Another is Telemetry (aka Insights, aka Cinnamon). Neither of these would open random ports, however, and neither would prevent cluster formation.
CodePudding user response：
It turned out that the configuration is correct. All I needed was
sudo iptables -Fto clear the IP Tables- Make the IP Tables persistent so they would not reset back to something else after every reboot.
Clearing IP Tables sounds like a giant security risk, but I already have my rules defined through Oracle's Virtual Cloud Network. Still, I don't know about the security risks that what I did involves.
As for arbitrary port numbers in the output of netstat -ano -p tcp, I think I was interpreting that incorrectly.
- First, I think in Akka config I cannot control client port numbers, only the server ones (client/server at the socket level).
- Second, in the output of
netstat -ano -p tcp, on every line for my java process, exactly one of the ports in Local Address and Foreign Address columns are what I have in the configuration. So I guess that is how I can figure out client/servers at the socket level in each of those lines (port numbers that I did put in the configuration are client side).