I have the following architecture.

I have a microservice that is subscribed to a Redis pub/sub, sometimes at the day the load will be high and I will need to make a horizontal pod autoscaler in K8S, in order to process all pub/sub messages while keeping the low latency.

The problem is that if I use pub/sub the message will be processed by each pod, and if use a queue the pods will need to make polling to Redis in order to read all messages, so, how can I scale that with pattern or approach I need to use or to know?

Thanks.

CodePudding user response：

If you want to have multiple instances of your microservice to process a different message that was published (in order to drain the queue faster or keep a low latency), then Redis pub/sub is not the right choice here, because it has a fan-out design, which means that the same published message will be dispatched to all subscribers.

What you need here is RabbitMQ or Amazon SQS, that allow you to have multiple consumers connected to the same queue, and each consumer is processing a different message.

A very useful feature in SQS, is that when a consumer takes a message, there is a configurable timeframe called visibility timeout, during which other consumers cannot see that message (and thus, process it). If the consumer that initially took the message does not remove it from the queue within that timeframe, then the message will be available to other consumers.