Can SqsFifoPartialProcessor circuit-break only on records with same group ID?

[duc00]

Hello,

According to Implementing partial batch responses AWS doc:

If you're using this feature with a FIFO queue, your function should stop processing messages after the first failure and return all failed and unprocessed messages in batchItemFailures. This helps preserve the ordering of messages in your queue.

I see that Powertools implementation of the doc, with SqsFifoPartialProcessor, is strictly following this recommendation. This question is thus more specific to AWS implementation of partial batch responses with FIFO. Posting the question on this repo seems to be a good entry point nonetheless, since both AWS developers and community interact on those subjects.

My problem is the following:
I am processing a SQS FIFO queue with SqsFifoPartialProcessor. My batch size is 10. Since the queue is not high-scale, the batch often contains messages with different message group IDs. When a failure occurs, the rest of the processing is stopped and all records left are returned as failures to the queue. So I often end-up with valid records in my dead-letter queue just because they were processed in a batch containing an unrelated invalid one.

My question:
Would it be valid, after a failure, to return all remaining records in the batch but only the ones with the same group ID? According to the doc, current implementation is recommended to preserve the ordering of messages in your queue. I am not seeing why processing other records with a different group ID would go against that. Thus my question.

Many thanks!

[leandrodamascena]

Hello @duc00! Sorry for the late reply, very busy days here. I'm assigning @rubenfonseca to answer this question because he created this processor and did some research on processing FIFO queues with partial failure.

[rubenfonseca]

Hi @duc00 thank you so much for opening this discussion, and for the clear explanation!

I've spent some time analysing the code and why we've implemented it that way, and it boils down to your original quote from the documentation:

If you're using this feature with a FIFO queue, your function should stop processing messages after the first failure and return all failed and unprocessed messages in batchItemFailures. This helps preserve the ordering of messages in your queue.

As stated above, the official AWS documentation doesn't mention that you can continue to process a different message group ID. It explicitly says to fail the rest of the messages.

This should not create any bug, since the rejected messages should be automatically retried by SQS. Having said that, I understand that there could be opportunities for optimization based on your comments. However, given the official documentation, and out of an abundance of caution, we believe the current implementation should be the safest.

Hope this helps, and thank you again for your great question!

[duc00]

Hi @rubenfonseca,

Thanks for the detailed answer. About:

This should not create any bug.

It does, as messages can be retried within new batches containing the failing (thus circuit breaking) message. As a consequence, the valid messages can end up in the dead-letter queue after max retries.

However, I understand that this is not a powertools implementation question, but one for AWS SQS team. I will direct it to a more appropriate channel.

Thanks!

[rubenfonseca]

@duc00 I want to dive deeper into this, and I've been trying to collect information. I've also talked to other customers using SQS FIFO, and they referenced this part of the documentation:

When you send a message to a FIFO queue, you specify a message group ID. Amazon SQS ensures that messages in the same group are delivered to Lambda in order. Lambda sorts the messages into groups and sends only one batch at a time for a group.

So I wanted to ask you: can you confirm me if you ever saw more than one message group ID on the same Lambda invocation when connected to a SQS FIFO queue?

[duc00]

@rubenfonseca yes, I can confirm that I saw more than one message group ID on the same Lambda invocation when connected to a SQS FIFO queue. This is expected behaviour, as described in the documentation:

It is possible to receive up to 10 messages in a single call using the MaxNumberOfMessages request parameter of the ReceiveMessage action. These messages retain their FIFO order and can have the same message group ID. Thus, if there are fewer than 10 messages available with the same message group ID, you might receive messages from another message group ID, in the same batch of 10 messages, but still in FIFO order.

[rubenfonseca]

Perfect! We pretty much confirmed this too. Here's our plan:

1. we'll write some sample code to try to emulate the behavior, to make sure we can test it
2. we'll try and test to continue to process other message group ids, to see if there isn't any bad consequences
3. we'll assess how this may or may not be a breaking change, and evaluate if this should be the default behavior or behind a feature flag

For now I'll convert this discussion into an issue! Thank you for your feedback so far :)

[duc00]

Great, thanks @rubenfonseca. I can help on this one if needed

[rubenfonseca]

Converted this into an issue: #2981