Enable Prometheus on multiple ActorSystems in the same JVM-CodePudding

I'm trying to monitor my Akka system with Prometheus and Grafana from Lightbend Telemetry. My configuration looks like

cinnamon {
akka.actors = {
default-by-class {
  includes = "/user/*"
  report-by = class
}
  }
  }
cinnamon.prometheus {
      exporters  = http-server
    }

With one ActorSystem everything works fine, but as soon as I instantiate another one I get the WARN and nothing works anymore.

[info] [WARN] [02/23/2022 11:47:30.051] [main] [PrometheusHttpServer] Could not start Prometheus HTTP server. Is there already a server running on this port? (This can happen when running multiple actor systems in the same JVM.)

I was wondering how to run Prometheus with multiple ActorSystems on the same JVM. Anyone can help me ?

CodePudding user response：

The problem is that cinnamon starts a separate http server for prometheus metrics and uses the same port when second actor system boots. The port is configured via cinnamon.prometheus.http-server.port but you cannot just change it in application.conf file because it will be shared between different actor systems and the port will be reused again.

You should just use a single actor system. There is no real reason of having multiple ones. However, if this is outside of your control, you could do the following for actor systems you start.

You need to change default behavior of loading configuration. One way to do it is following.

import akka.actor.ActorSystem
import com.typesafe.config.{Config, ConfigFactory}
import scala.jdk.CollectionConverters._

def overriddenConfig(map: Map[String, String]): Config = {
  ConfigFactory.parseMap(map.asJava).withFallback(ConfigFactory.load())
}

Now you can start actor systems in following way

ActorSystem("system1", overriddenConfig(Map("cinnamon.prometheus.http-server.port" -> "9091")))
ActorSystem("system2", overriddenConfig(Map("cinnamon.prometheus.http-server.port" -> "9092")))

The other option is to split configuration files for each actor system and each actor system would load from different file. I will leave the code out but it can be done via ConfigFactory methods.

Also please note that this approach will require Prometheus to read metrics from multiple http endpoints, each representing separate actor system.