Azure Service Bus, using filters to assemble a large message broken up into smaller messages-CodePudding

I'm trying to find a solution for receiving large messages on Azure Service Bus. The essential pattern I was thinking is to publish a large messages in parts -- along with a correlation id, a page, and an "of".

So if I have a four-part message, they would all have the same correlation id, each would have an "of" of 4, and the page would be 0 - 3. The set would be published as a batch.

The listener could listen for only messages with a page of 0, and then pull the remaining messages according to the transaction id.

Publishing these messages is easy enough. ServiceBusMessage has a CorrelationId field, and a dictionary field called ApplicationProperties that I can add my custom "page" and "of" fields to. I can assemble them into a ServiceBusMessageBatch before publishing.

What I'm not sure about is how to receive the messages. I'm using Function Apps, so it's easy to setup a listener.

[FunctionName("GeneralLogger")]
public static void Run([ServiceBusTrigger("queueName", Connection = "AzureWebJobsServiceBus")] string myQueueItem, ApplicationProperties ap, ILogger log)
{ /// process message }

But I don't see how to filter here. Also, I can pull messages by adding a handler to the message processor, described here: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues But likewise I don't see how to filter.

The only Azure Service Bus filtering I see how to do is between a topic and subscription. There is a lot of capability there, but nothing dynamically I can set during runtime.

I feel like I'm either trying to miss-use something or re-inventing the wheel. Is anyone else doing something like this with Azure Service Bus?

CodePudding user response：

It isn't possible to apply filters on a queue; they only operate on topics/subscriptions.

Generally, the Claim Check pattern is recommended when you're looking to send a payload too large for a single message. In a nutshell, you would write your payload to some form of durable storage and then your Service Bus message would provide the location for consumers.

An example implementation using the Azure.Messaging.ServiceBus package can be found in this sample.

CodePudding user response：

I'm trying to find a solution for receiving large messages on Azure Service Bus.

A solution is already there. It's Azure Service Bus premium tier. Capable of sending messages up to 100MB in size. It comes with a price. Assuming you're looking to spit up the file either because the premium is much to pay for or because messages could be larger than 100MB, the claim-check pattern is the way to go. There's just one issue when the claim-check pattern is used over the premium tier - you cannot have a deterministic clean-up when a message is an event, and there are multiple receivers. You'd need to come up with some policy to clean up those blobs, given that those are large blobs and will quickly add to the storage consumption over time, depending on the number of messages flowing through the system. With the premium tier, the problem of clean-up doesn't exist. Nor do you have to provide a storage account. Therefore, if your large messages will not exceed 100MB, it could be a more suitable solution for your production environment.