[server] Recover log and index file for unclean shutdown #1749
Conversation
LiebingYu
force-pushed
the
fix-corrupt-index-new
branch
2 times, most recently
from
b9b5f2d to
ceaf704
Compare
|
Ready for CR @wuchong @swuferhong |
| new WriterStateManager( | ||
| logSegments.getTableBucket(), | ||
| logTabletDir, | ||
| this.writerStateManager.writerExpirationMs()); |
There was a problem hiding this comment.
Why we need to new a WriterStateManager? Maybe we can add a clear() method?
There was a problem hiding this comment.
This logic follow Kafka, and ceate a new WriterStateManager is a lightweight operation. I think it's ok.
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
| numUnflushed, | ||
| logSegments.getTableBucket()); | ||
|
|
||
| int truncatedBytes = -1; |
There was a problem hiding this comment.
Do we need to be this too aggressive here? Deleting all subsequent logSegments just because one cannot be repaired — I feel this might pose a risk of data loss. Also, we don't have test coverage for this logic.
There was a problem hiding this comment.
This logic also follow Kafka. From my perspective, data loss is unlikely because the data is stored in multiple replicas. Once the file is truncated to the correct position, it can synchronize the latest data from the leader. If truncation is not carried out, the file appears to be unrecoverable, and if the host machine becomes the leader afterward, unforeseen problems might occur.
There was a problem hiding this comment.
And I add test to cover this.
LiebingYu
force-pushed
the
fix-corrupt-index-new
branch
from
ceaf704 to
1e8a582
Compare
Purpose
Linked issue: close #1716
Brief change log
Tests
API and Format
Documentation