Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IOTDB-6355] Fix query scan will return duplicated timestamp or unordered timestamp while TsFileResource degrading #14458

Merged
merged 1 commit into from
Dec 17, 2024

Conversation

JackieTien97
Copy link
Contributor

@JackieTien97 JackieTien97 commented Dec 17, 2024

Assuming that we have the following file distribution, and all the files shown have been degraded to FileTimeIndex:
image

While we do scanning, we will firstly unpack the frist seq file and the first and second unseq file, then while we merge on read, we will continue to get timestamp 4, then we using startTime of the second seq time which is 5 to judge whether it's overlapped with timestamp 4. 5 is larger than 4, so we think that the second seq tsfile isn't overlapped with timestamp 4, and as it's in seq space, so we mistakenly assumend that none of the subsequent files would overlap with timestamp 4.

The above processing is correct while there doesn't exist degraded resource in seq space, however while exsiting degraded resource in seq space, if current seq tsfile's startTime is larger than the current timestamp, we cannot conclude that the startTime of subsequent seq tsfiles are all larger than current timestamp.

The solution is that we will return Long.MIN_VALUE for FileTimeIndex in seq space which means that all the degraded seq resources will be unpacked anyway. And then we can use the TimeSeriesMetadata to further precisely judge we need to stop searching:

asc: if timeseries metadata of current seq tsfile's endTime >= endpointTime, we don't need to continue
desc: if timeseries metadata of current seq tsfile's startTime <= endpointTime, we don't need to continue

Copy link

sonarcloud bot commented Dec 17, 2024

Copy link

codecov bot commented Dec 17, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 39.62%. Comparing base (6a28a07) to head (e9768b1).
Report is 5 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14458      +/-   ##
============================================
- Coverage     39.62%   39.62%   -0.01%     
  Complexity      185      185              
============================================
  Files          4331     4331              
  Lines        275619   275632      +13     
  Branches      33497    33503       +6     
============================================
  Hits         109226   109226              
- Misses       166393   166406      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants