Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flink] Fix query service error while query table with deletion vectors option #4301

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Shadowell
Copy link
Contributor

Purpose

Test query service with deletion vectors options, its okay for table with option 'deletion-vectors.enabled' = 'true'

Linked issue: close #4265

Tests

Add test testQueryServiceWithDeletionVectors

API and Format

No

Documentation

No

@herefree
Copy link
Contributor

In my test,CALL sys.query_service('default.DIM', 1) can be submitted to the flink cluster,but it will always restart due to exceptions.I don’t know if you have actually tested it on the flink cluster.

java.lang.IllegalStateException: SortedRun is not sorted and may contain overlapping key intervals. This is a bug.
at org.apache.paimon.utils.Preconditions.checkState(Preconditions.java:182)
at org.apache.paimon.mergetree.SortedRun.validate(SortedRun.java:90)
at org.apache.paimon.mergetree.SortedRun.fromUnsorted(SortedRun.java:67)
at org.apache.paimon.mergetree.Levels.updateLevel(Levels.java:190)

@herefree
Copy link
Contributor

a2926d01-d72a-4229-955f-f8a32024c537
d33d21da-49e0-42a8-aa2c-cda53265e0a2

@Shadowell Shadowell changed the title [flink] Add query service test with deletion vectors options [WIP][flink] Add query service test with deletion vectors options Oct 12, 2024
@Shadowell Shadowell changed the title [WIP][flink] Add query service test with deletion vectors options [flink] Fix query service error while query table with deletion vectors option Oct 24, 2024
@Shadowell
Copy link
Contributor Author

Shadowell commented Oct 25, 2024

Hi @herefree ,I test it in local standalone cluster, it does throw this exception. Due to add option 'deletion-vectors.enabled' = 'true', the file level is upgrade from level-0 to level-5 duration compaction. So duration process element, the file was added to sortRun file again which lead to file duplicate and that cause validate sortRun failed.
cc @JingsongLi

Add I find another bug duration test.
I only insert one record, but query service show process 2 records.
And I will raise another issue to follow this.
image
image

@herefree
Copy link
Contributor

Hi @herefree ,I test it in local standalone cluster, it does throw this exception. Due to add option 'deletion-vectors.enabled' = 'true', the file level is upgrade from level-0 to level-5 duration compaction. So duration process element, the file was added to sortRun file again which lead to file duplicate and that cause validate sortRun failed. cc @JingsongLi

Add I find another bug duration test. I only insert one record, but query service show process 2 records. And I will raise another issue to follow this. image image

Thanks your reply, look forward to your repair~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] when table set deletion-vectors.enabled =true, call query_service error.
2 participants