Does Amazon SQS FIFO provide message ordering?-CodePudding

I am trying to implement AWS SQS FIFO queue using spring boot(v2.2.6.RELEASE).

Created a queue, "Testing.fifo" in aws. Left all other fields to default while creating the queue.

My producer and consumer to the queue run on a single service.

code to put messages to queue

private void sendMessageToSqsFiFo(String queueName, Object obj, String messageGroupId, String messageDeduplicationId) {
    // TODO Auto-generated method stub
    if (messageGroupId == null) {
        // setting a default group
        messageGroupId = "default_group_id";
    }
    if (messageDeduplicationId == null) {
        // setting unique deduplication id for all messages
        messageDeduplicationId = "sqs-fifo-deduplication-"  UUID.randomUUID();
    }
    Map<String, Object> headers = new HashMap<>();
    headers.put("message-group-id", messageGroupId);
    headers.put("message-deduplication-id", messageDeduplicationId);
    queueMessagingTemplate.convertAndSend(queueName, obj, headers);
    System.out.println("Message sent to FIFO "  queueName);
}

Queue listener code

@SqsListener(value = "Testing.fifo", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
    private void getMessageFromgeneralQdasd(Map<String, String> msg, @Header("MessageId") String messageId, Acknowledgment ack) {
         System.out.println("Message from FIFO: "   msg.toString());
         ack.acknowledge();
}

I am putting some messages to queue on service start up

@EventListener(ApplicationReadyEvent.class)
        public void doSomethingAfterStartup() {
            System.out.println("hello world, I have just started up");      
            for(int i=0;i<25;i  ) {
                 Map<String, String> data = new HashMap<String, String>();
                 data.put("message", i   "th msg");
                sendMessageToSqsFiFo("Testing.fifo", data, null, null);
            }
        }

I am expecting a sysot log like this

Message from FIFO: {message=0th msg}
Message from FIFO: {message=1th msg}
Message from FIFO: {message=2th msg}
Message from FIFO: {message=3th msg}

......
Message from FIFO: {message=24th msg}

in the order I pushed data to queue

But I got

Message from FIFO: {message=0th msg}
Message from FIFO: {message=2th msg}
Message from FIFO: {message=3th msg}
Message from FIFO: {message=1th msg}
Message from FIFO: {message=4th msg}
Message from FIFO: {message=6th msg}
Message from FIFO: {message=7th msg}
Message from FIFO: {message=5th msg}
Message from FIFO: {message=8th msg}
Message from FIFO: {message=9th msg}
Message from FIFO: {message=10th msg}
Message from FIFO: {message=11th msg}
Message from FIFO: {message=12th msg}
Message from FIFO: {message=13th msg}
Message from FIFO: {message=14th msg}
Message from FIFO: {message=15th msg}
Message from FIFO: {message=16th msg}
Message from FIFO: {message=17th msg}
Message from FIFO: {message=19th msg}
Message from FIFO: {message=18th msg}
Message from FIFO: {message=20th msg}
Message from FIFO: {message=21th msg}
Message from FIFO: {message=22th msg}
Message from FIFO: {message=23th msg}
Message from FIFO: {message=24th msg}

Am I doing anything wrong.?

using dependency aws-java-sdk 1.11.586

CodePudding user response：

This might help: The message group ID is used for ordering of SQS messages in groups.

So, if you have multiple group IDs like 1 and 2, then within those groups, the SQS messages will always be in order. The ordering will be preserved relative to the group and not the queue as a whole.

This can be used in cases like, if you are sending out events for different user details updates, you would want to group the data for a particular user in order, and at the same time you don't care about the ordering of the queue as a whole. You can then set the group ID as the unique user ID in this case.

If there is no use-case of creating groups, and there is only one group and you want the message order to be preserved as a whole, just use the same groupID for all messages.

more info here: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html

CodePudding user response：

One reason could be that your SQSListener is run on a thread pool created by default. Since different threads pick up the message from the ordered queue but post that it's all depends upon thread scheduling so basically you cannot guarantee order.If there is more than one thread processing on the consumer side you get the benefit of horizontal scaling but cannot guarantee order. If the thread is single in the threadpool then it will be in order. Similarly if there are multiple instances of consumer running ( even though each consumer is having single thread) again there will be no guarantee of order. You can verify this by printing the thread id/name in your program which can tell if multiple threads are processing.