Another overload of distinct

[julianhowarth]

We are using Mutiny with a streaming process reading data from AWS DynamoDB, doing some transformations and then writing to S3. Doing it this way means the memory footprint is low as we never need to hold the entire dataset in memory.

We recently discovered we were getting duplicates in the dataset and so turned to Multi's select().distinct(comparator) operator in order to remove them. This caused the memory usage to balloon as now the entire dataset has to be held in memory for duplicate checking. However, our duplicate check is trivial, based on an integer id only, so there is no need for the entire dataset to be in memory, just a Set<Integer>.

We've been able to solve this very easily by using plug with a custom AbstractMultiOperator which uses an extractor instead of a Comparator to extract the id from each of our objects and store that in a Set rather than the objects themselves. Doing this has returned the memory footprint to its previous low value.

Would it be useful to have this directly in Mutiny? So to add a new method to MultiSelect, something like:

public <K> Multi<T> distinct(Function<T, K> keyExtractor) {

where keyExtractor is responsible for extracting a key which is used to de-duplicate the values. This could be implemented as a separate AbstractMultiOperator, something like MultiDistinctByKeyOp<T, K>

[cescoffier]

Sounds reasonable, so +1.
Fancy a PR?

@jponge in on PTO this week, but will be able to assist you next week.

[julianhowarth]

Thanks, I'll put together a PR over the next few days

[cescoffier]

Awesome! Thanks!

[julianhowarth]

I have raised #1731 as a first attempt at addressing this.

[jponge]

Implemented in #1731