Aggregate grouping

[vasco-santos]

Today we create aggregates without any grouping, while the underneath protocol https://github.com/web3-storage/specs/blob/main/w3-filecoin.md enables us to perform grouping by storefront and groups within a storefront (for instance, free version vs paid version).

We were hoping to rely on SQS Message Group ID to do this, which is not possible because:

1. only FIFO SQS Queues support Message Group ID
2. message Group ID actually behaves differently than expected. It does not invoke Queue consumers with batches of a single group, but only guarantees order of messages being delivered within a group

By current priorities, we will not need an implementation of aggregate grouping at this point.

When we want to implement this, a few options were already discussed:

• Having provisioned queues per product
  • while likely more efficient and cost optimal, it would require manual provisioning which is not ideal
• Iterate on the Buffer data structure from an array like to a map like where we can keep track of multiple groups at the same time

[Gozala]

#48 dropped this given that supporting multiple storefronts is no longer a requirement. Yet I still think we should design system such that would allow that requirement to change again, if it does not introduce significant complexity.

In this specific instance we could easily accommodate multiple tenants simply by changing our buffer data structure from T[] to Record<T['groupId'], T[]>.

Argument could be made that such change could be implemented when requirement arises, while true, it appears to me that making that change now would be trivial while making while pipeline is active would introduce some challenges.

Separately it also might be a good idea to tag our buffer data structure so it can be versioned e.g { "aggregate/[email protected]": T[] } that way we'll be able to introduce { "aggregate/[email protected]": Record<T['groupId'], T[]> } in the future.