I am new to nginx, and am wondering if it can help me to solve a use-case we've encountered.

I have n nodes,which are reading from from a kafka topic with the same group id, which means that each node has disjoint data, partitioned by some key.

Nginx has no way of knowing apriori which node has data corresponding to which keys. But we can build an API or have a redis instance which can tell us the node given the key.

Is there a way nginx can incorporate third party information of this kind to route requests?

I'd also welcome any answers, even if it doesn't involve nginx.

CodePudding user response：

Nginx has no way of knowing apriori which node has data corresponding to which keys

Nginx doesn't need to know. You would need to do this in Kafka Streams RPC layer with Interactive Queries. (Spring-Kafka has an InteractiveQueryService interface, btw, that can be used from Spring Web).

If you want to present users with a single address for the KStreams HTTP/RPC endpoints, then that would be a standard Nginx upstream definition for a reverse proxy, which would route to any of the backend servers, which in-turn communicate with themselves to fetch the necessary key/value, and return the response back to the client.

I have no idea how Kafka partitions

You could look at the source code and see it uses a murmur2 hash, which is available in Lua, and can be used in Nginx.

But again, this is a rabbit hole you should probably avoid.

Other option, use Kafka Connect to dump data to Redis (or whatever database you want). Then write a very similar HTTP API service, then (optionally) point Nginx at that.