Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pauseless Consumption #3: Disaster Recovery with Reingestion #14920

Merged
merged 70 commits into from
Feb 12, 2025
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
6defce3
Controller side changes to allow pauseless ingestion:
9aman Dec 19, 2024
54ab7b3
Minor improvements to improve readability
9aman Dec 20, 2024
1e40134
Server side changes to enable pauseless consumption. The changes include
9aman Dec 23, 2024
012da87
Fixes in the commit protocol for pauseless ingestion to complete the …
9aman Dec 24, 2024
a97847f
Changing the way to enable pausless post introduction of multi-stream…
9aman Dec 30, 2024
2c2ba86
WIP: Changes in the expected state while performing recovery during p…
9aman Dec 30, 2024
4d7c893
Adding changes for ensurePartitionsConsuming function that is utilize…
9aman Dec 30, 2024
a041a75
Add Server side Reingestion API
Dec 30, 2024
b6d0904
run segment level validation
Dec 30, 2024
58f6c51
Add method to trigger reingestion
Dec 30, 2024
d6313b3
Linting fixes
Dec 30, 2024
ca6134a
Adding integration tests for 3 failure scenarios that can occur durin…
9aman Jan 2, 2025
845f616
WIP: Reingestion test
Jan 6, 2025
d2dd313
Fix bug with indexes in reingestion
Jan 6, 2025
fb34fc8
Add tests for reingestion
Jan 8, 2025
50725bd
Fix controller url in Reingestion
Jan 14, 2025
e74d360
Support https and auth in reingestion
Jan 14, 2025
ce3c851
Merge branch 'master' into resolve-failures-pauseless-ingestion
9aman Jan 15, 2025
d6208a6
Removing checks on default crc and replacing them with segment status…
9aman Jan 15, 2025
7f5b720
Formatting improvements
9aman Jan 15, 2025
3f05b2f
Allowing null table config to be passed for checking pauseless consum…
9aman Jan 15, 2025
2012e38
Ensuring upload retries are backward compatible
9aman Jan 15, 2025
7b9da37
Removing unnecessary code
9aman Jan 15, 2025
ded8962
Fixing existing test cases and adding unit tests to check upload to d…
9aman Jan 16, 2025
e703d84
Refactor test cases to reduce repetition
9aman Jan 16, 2025
c2fda4a
Add missing header file
9aman Jan 16, 2025
c836009
Fix reingestion test
Jan 17, 2025
7974ab9
refactoring file upload download client
Jan 17, 2025
c08f841
Removing pauselessConsumptionEnabled from index config
9aman Jan 21, 2025
c4b99bd
Remove reingestion code
Jan 21, 2025
aee514c
Removing check in replaceSegmentIfCrcMismatch
9aman Jan 23, 2025
88a619a
Adding a new class for simple serialization and deserialization of ma…
9aman Jan 24, 2025
791ac21
Removing files related to reingestion tests
9aman Jan 24, 2025
8db5bae
Merging master and including force commit PR changes
9aman Jan 24, 2025
55b2b29
Revert "Remove reingestion code"
Jan 27, 2025
f74df66
Revert "Removing files related to reingestion tests"
Jan 27, 2025
1f4db11
Fix reingestion issue where consuming segemnts are not replaced
Jan 27, 2025
8e9249c
Copy full segment to deep store before triggerring metadata upload
Jan 27, 2025
b804a69
Refactoring: added support for tracking running reingestion jobs
Jan 27, 2025
609942d
Refactor: fix doc comments
Jan 27, 2025
f939714
Make SegmentZKMetadata JSON serializable
Jackie-Jiang Jan 28, 2025
155c49f
Minor API name change
Jackie-Jiang Jan 28, 2025
080ec55
Refactor PinotLLC class to add ability to inject failures
Jan 28, 2025
7e04fa3
Minor improvements
9aman Jan 28, 2025
523913f
Moving ZkMetadaptaUtils to commons and reusing the code in the upload…
9aman Jan 28, 2025
5689333
Fix lint failures
Jan 28, 2025
f42c6b8
Misc fix and cleanup
Jackie-Jiang Jan 29, 2025
a2eebf9
The tests were running slow due to the condition that the IdealState …
9aman Jan 29, 2025
1ba5b6c
Refactor Reingestion integration test
Jan 29, 2025
11aa170
Merge remote-tracking branch 'upstream/master' into pauseless-reinges…
Jan 29, 2025
e84788a
Fix error in tests post rebase
Jan 29, 2025
0d46327
refactoring
Jan 30, 2025
a94c7e3
Remove redundant code
Feb 3, 2025
8b9b8d1
Add support for queue in reingestion
Feb 3, 2025
d035249
Refactoring
Feb 4, 2025
bc8a65b
refaactoring
Feb 4, 2025
f082d24
Honour segment build semaphore during reingestion
Feb 4, 2025
d1ad30b
Fix test
Feb 5, 2025
7e47dd5
Add a seperate API to upload reingested segments
Feb 5, 2025
5a42c28
Cleanup code
Feb 5, 2025
1551685
Replace reIngest with reingest
Feb 6, 2025
837aa26
Replace reIngest with reingest
Feb 6, 2025
09e8583
Ensure correctness in reingestion
Feb 6, 2025
3d6fdf4
Decouple reingest completion and upload ZK path
Feb 6, 2025
b525c7e
Refactor Stateless segment writer to have least number of arguments
Feb 6, 2025
f7ae25f
Fix segment reset for pauseless tables
Feb 6, 2025
3c17957
Cleaning up Stateless segment writer
Feb 6, 2025
a3fa25c
Rename metric
Feb 6, 2025
61ebb23
Misc cleanup and logic decoupling
Jackie-Jiang Feb 12, 2025
62316c0
nit
Jackie-Jiang Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactoring file upload download client
KKCorps committed Jan 17, 2025
commit 7974ab926d6305f6cee721ec62db6cfc5263a627
Original file line number Diff line number Diff line change
@@ -1301,25 +1301,13 @@ public void triggerReIngestion(String serverHostPort, String tableNameWithType,

URI reIngestUri = getURI(scheme, serverHost, Integer.parseInt(serverPort), REINGEST_SEGMENT_PATH);

// Build the JSON payload
Map<String, Object> requestJson = new HashMap<>();
requestJson.put("tableNameWithType", tableNameWithType);
requestJson.put("segmentName", segmentName);

// Convert the request payload to JSON string
String jsonPayload = JsonUtils.objectToString(requestJson);
// Prepare a POST request with Simple HTTP
ClassicRequestBuilder requestBuilder = ClassicRequestBuilder
.post(reIngestUri)
.setVersion(HttpVersion.HTTP_1_1)
.setHeader("Content-Type", "application/json")
.setHeader("Accept", "application/json")
.setEntity(new StringEntity(jsonPayload, ContentType.APPLICATION_JSON));

// Send the request using your custom HttpClient wrapper.
// (Adjust the timeout as needed in your environment)
SimpleHttpResponse response = HttpClient.wrapAndThrowHttpException(
_httpClient.sendRequest(requestBuilder.build(), HttpClient.DEFAULT_SOCKET_TIMEOUT_MS));
SimpleHttpResponse response =
HttpClient.wrapAndThrowHttpException(_httpClient.sendJsonPostRequest(reIngestUri, jsonPayload));

// Check that we got a 2xx response
int statusCode = response.getStatusCode();