[FLINK-38464] Introduce OrderedMultiSetState #27071
Conversation
to be able to find serializer tests
|
|
||
| /** | ||
| * Remove the given element. If there are multiple instances of the same element, remove the | ||
| * first one in insertion order. |
There was a problem hiding this comment.
I am curious : should we allow the user to choose LIFO or FIFO for the remove ?
| * The most recently added element was removed. The result will contain the element added | ||
| * before it. | ||
| */ | ||
| REMOVED_LAST_ADDED, |
There was a problem hiding this comment.
I don't see tests checking these removal result types.
| * An element was removed, it was not the most recently added, there are more elements. The | ||
| * result will not contain any elements | ||
| */ | ||
| REMOVED_OTHER |
There was a problem hiding this comment.
I am curious why nothing is returned in this case, this seems inconsistent with REMOVED_LAST_ADDED which will return the element added before it.
| SizeChangeInfo append(T element, long timestamp) throws Exception; | ||
|
|
||
| /** Get iterator over all remaining elements and their timestamps, in order of insertion. */ | ||
| Iterator<Tuple2<T, Long>> iterator() throws Exception; |
There was a problem hiding this comment.
if the multi set changes (i.e. there is a removal) under the iterator what will happen? AI unit test for this would be good. It would be useful to understand any locking that has been considered or is in place.
| @Internal | ||
| @Experimental |
There was a problem hiding this comment.
🤔 aren't those two mutually exclusive? Class can either be Internal and not intended to be used outside of the Flink's repo, or Experimental/PublicEvolving/Public depending on the stability of the said api
There was a problem hiding this comment.
I don't see why @Experimental implies not-internal but I'd be fine removing it.
There was a problem hiding this comment.
Here you have stated that Experimental is intended to be used by users
so it has to be dropped here :)
There was a problem hiding this comment.
Thanks for the pointer, dropped @Experimental.
...n/java/org/apache/flink/table/runtime/orderedmultisetstate/AdaptiveOrderedMultiSetState.java
Outdated
Show resolved
Hide resolved
...n/java/org/apache/flink/table/runtime/orderedmultisetstate/AdaptiveOrderedMultiSetState.java
Outdated
Show resolved
Hide resolved
...main/java/org/apache/flink/table/runtime/sequencedmultisetstate/ValueStateMultiSetState.java
Show resolved
Hide resolved
...c/main/java/org/apache/flink/table/runtime/orderedmultisetstate/ValueStateMultiSetState.java
Outdated
Show resolved
Hide resolved
github-actions
bot
added
the
community-reviewed
.../src/main/java/org/apache/flink/table/runtime/orderedmultisetstate/OrderedMultiSetState.java
Outdated
Show resolved
Hide resolved
...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java
Outdated
Show resolved
Hide resolved
...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java
Outdated
Show resolved
Hide resolved
...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java
Show resolved
Hide resolved
| * @see org.apache.flink.api.common.state.ValueState | ||
| */ | ||
| @Internal | ||
| public class LinkedMultiSetState implements OrderedMultiSetState<RowData> { |
There was a problem hiding this comment.
I think we should have test coverage for this quite complex class. Ditto for Value and Adaptive versions, so probably one test suite to be executed on different implementations with adaptive configured in a couple of different ways?
Plus some dedicated tests for switchover in Adaptive version?
There was a problem hiding this comment.
Definitely, so far I was adding the tests at the operator level, but will add the tests in this PR as well.
There was a problem hiding this comment.
I've added the tests (and fixed the found bugs).
Introduce
OrderedMultiSetStateand 3 implementations (map, value, adaptive) to be used in SInkUpsertMaterializerV2.Test coverage is currently provided on the operator level (#27070).
I'm planning to add lower-level unit tests later to this PR.