I'm building a data flow pipeline for cached database queries for multiple users with Kafka and Flink etc. and I am confused about when I need to implement multi threading.

I have a built a user service that contains an HTTP server using Vertx. I have an endpoint that accepts some input from users. One of the endpoints makes a network request to a Kafka service and returns some information from a topic to the user. I created a "kafka client" for this, and in every user request to the endpoint I create a new instance of this kafka client to send the request. Every user's request is totally unrelated, so it doesn't matter whose request gets processed first - there's no race condition to worry about.

1. If I create a new instance of this kafka client for every incoming user request, an exception thrown in user 1's request should not block user 2's request right? (because I'm already in a separate user thread)?

2. Is creating a new instance of a client on every request a good practice? I'm used to creating singletons and reusing them in Node projects (not sure how you even "import" classes in Java).

3. Instead of creating new instances on every request, if I create one singleton client and import it into the endpoint so that it gets reused on every user's request, would an exception thrown in user 1's request then block user 2's request? (Unless I manually added multithreading) or if each user gets their own thread, an exception in user 1’s thread would not affect user 2‘s thread even if they’re using the same singleton?

CodePudding user response：

In vert.x you typically try to avoid implementing multithreading yourself. You should simply create your handlers and vert.x will call them when they are needed. You can read more about how this works in vert.x here.

1. Thrown exceptions don't block any request, not even the one where they are thrown. They will however cause the request to fail, if you forget to handle them. The exception would not affect the request of the second user at all, because they are isolated from each other. Note that this also depends on the nature of the exception. If you share your kafka client and it gets disconnected, then all users will get the same error individually.

2. That completely depends on the client that you are creating and you should check out their documentation. You can also have a look at connection pooling if you're interested in this topic. You can try creating the client inside of your class, instead of creating a new one for each request. You can also use the vert.x kafka client and create it inside your verticle's start function. This way you should be on the safe side.

3. Same as 1.