Basically I'm builing a system on google cloud. Most services are working on k8s cluster but some codes are not. Lambda and operator of composer, dataflow job are the examples. (Composer is also k8s but different cluster)

I picked kafka as event channel to interconnect the services and I have to decide proper place of kafka broker. K8s pods or VM. I prefer k8s pods, but I worry about the communication between brokers and services, espicially with services outside of k8s cluster.

Consumer addresses broker with "bootstrap server" that is list of some broker's static unique address. I suppose if brokers are installed inside k8s, addresses of them will be not static unique from outside. Can brokers are connected from service outside of k8s? If possible, which string must be provided to bootstrap sever config?

Conventional virtual machine is the solution without any suspicion. But I want put more and more things into k8s.

CodePudding user response：

There are a different solutions to your problems

You can deploy the Kafka on K8s cluster and use the service mesh to interconnect both clusters. So broker and service can connect with each other without any worry.

If you are on GCP you can use the MCS service or traffic director and other service mesh.

You can also set up Kafka on VM and expose it over the IP and further that will be used by services to connect.

Can brokers are connected from service outside of k8s?

Yes, you can expose your Kafka broker using the service type Loadblanacer or Node Port. Reference doc

I suppose if brokers are installed inside k8s, addresses of them will be not static unique from outside.

You dont need to bind Kafka to any specific hostname for the interface, Kafka will listen to all the interfaces and you can expose it using the K8s service if running on K8s.