[controller] log compaction #1282

WhitneyDeng · 2024-11-05T00:00:12Z

Summary

purpose: to get stores ready for compaction based on programmatically defined criteria
added API endpoints to get compaction-ready stores

How was this PR tested?

unit test: filter all stores on a cluster for compaction-ready stores

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.
Yes. Make sure to explain your proposed changes and call out the behavior change.

…er methods VeniceParentHelixAdmin::isCompactionReady VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

…or readability & maintainability

…ForCompaction & VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours JavaDocs description - complete description of VeniceParentHelixAdmin::getStoresForCompaction - future tense to present tense for VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

…ionReady & helper methods

…ersion is not found

…:filterStoresForCompaction - extract filterStoresForCompaction() logic for testing - write VeniceParentHelixAdminTest::testFilterStoresForCompaction

… TestVeniceParentHelixAdmin - migrate from integration test files to unit test files

… argument order

mynameborat

Good job on putting up the first PR quickly. Looks very good. I have put in some comments.

Let me know if you have questions.

internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/ControllerClient.java

...mmon/src/integrationTest/java/com/linkedin/venice/controller/VeniceParentHelixAdminTest.java

services/venice-controller/src/main/java/com/linkedin/venice/controller/Admin.java

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

.../venice-controller/src/main/java/com/linkedin/venice/controller/server/AdminSparkServer.java

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

...nice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java

…tionReady package private

…data type ArrayList -> List

- remodularise VeniceParentHelixAdmin::isHybridStore to function var in VeniceParentHelixAdmin::isCompactionReady - make VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours hardcoded int to class constant

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

- change exception message in child controller

interface for repush: Azkaban/Airflow implementation for Linkedin internal. (potentially) Spark implementation for OSS.

…iceController

…logic to VeniceHelixAdmin

- empty implementation for now

reason: since repush will be used for diff use cases (other than compaction), the term repush should be reserved for the abstracted repush action.

- instantiate in VeniceController

- this config determines the number of threads for LogCompactionService::executor

…UNT annotations & comments

…into CompactionManager - create CompactionManager & TestCompactionManager - move TestCompactionManager::testFilterStoresForCompaction from TestVeniceParentHelixAdmin

…utor setup in LogCompactionService::startInner

…g time unit from hour to millisecond

internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java

...ntegrationTest/java/com/linkedin/venice/endToEnd/logcompaction/TestLogCompactionService.java

services/venice-controller/src/main/java/com/linkedin/venice/controller/Admin.java

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

...troller/src/main/java/com/linkedin/venice/controller/logcompaction/LogCompactionService.java

...controller/src/main/java/com/linkedin/venice/controller/repush/RepushOrchestratorConfig.java

...roller/src/test/java/com/linkedin/venice/controller/logcompaction/TestCompactionManager.java

...ler/src/test/java/com/linkedin/venice/controller/logcompaction/TestLogCompactionService.java

internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java

reason: not all repush orchestrators have execution UI url, but they all have execution ID

- RepushJobRequest data model for repush job params - move triggerSource logging from LogCompactionTask to RepushJobRequest. reason: LogCompactionService is upstream from manual & scheduled common path - add source region as optional repush job param

- delete RepushOrchestratorConfig, pass configs with VeniceProperties

…utor

- isCompactionReady() needs to do real call

lusong64

Whitney had an offline conversation about the latch thing and we resolved it.

Whitney Deng added 10 commits October 29, 2024 11:52

[controller] log compaction

ff3c1ae

[controller][server] complete log compaction candidate selection help…

6a7903d

…er methods VeniceParentHelixAdmin::isCompactionReady VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

[controller][server][style] put log compaction criterion on newline f…

62fcf64

…or readability & maintainability

[controller][server][documentation] VeniceParentHelixAdmin::isCompact…

8101408

…ionReady & helper methods

[controller][server][documentation] filter out stores where current v…

8160e5b

…ersion is not found

[controller][server][unit test] Unit test for VeniceParentHelixAdmin:…

c28728c

…:filterStoresForCompaction - extract filterStoresForCompaction() logic for testing - write VeniceParentHelixAdminTest::testFilterStoresForCompaction

[controller][server][unit test] move testFilterStoresForCompaction to…

5e481cb

… TestVeniceParentHelixAdmin - migrate from integration test files to unit test files

[controller][server][unit test][style] switch actual & expected value…

2a0f3f2

… argument order

[controller][server][unit test][style] switch actual & expected value…

f6ca080

… argument order

mynameborat reviewed Nov 6, 2024

View reviewed changes

Whitney Deng added 5 commits November 6, 2024 11:38

[controller][server][style][PR] make VeniceParentHelixAdmin::isCompac…

bf73b48

…tionReady package private

[controller][server][style][PR] revert wildcard import

77d83c0

[controller][server][style][PR] delete wildcard import

af12106

[controller][server][PR] change Admin::getStoresForCompaction return …

e82e659

…data type ArrayList -> List

[controller][server][PR]

373c208

- remodularise VeniceParentHelixAdmin::isHybridStore to function var in VeniceParentHelixAdmin::isCompactionReady - make VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours hardcoded int to class constant

majisourav99 reviewed Nov 7, 2024

View reviewed changes

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java Outdated Show resolved Hide resolved

Whitney Deng added 13 commits November 21, 2024 09:10

[controller]

2b11f9d

- change exception message in child controller

[controller] create RepushOrchestratorProvider

1a7f015

interface for repush: Azkaban/Airflow implementation for Linkedin internal. (potentially) Spark implementation for OSS.

[controller] create LogCompactionService & initialise instance in Ven…

1df10d9

…iceController

[controller][PR] change ArrayList to List datatype in contracts

4652dc9

[controller][PR] move VeniceParentHelixAdmin::getStoresForCompaction …

da5455c

…logic to VeniceHelixAdmin

[controller] add triggerRepush endpoint

f30f705

- empty implementation for now

[controller] change repush terminology to compactStore

24c97a1

reason: since repush will be used for diff use cases (other than compaction), the term repush should be reserved for the abstracted repush action.

[controller] create LogCompactionService

3991b22

- instantiate in VeniceController

[controller][config] add SCHEDULED_LOG_COMPACTION_THREAD_COUNT config

43f5c40

- this config determines the number of threads for LogCompactionService::executor

[controller][config][style] delete SCHEDULED_LOG_COMPACTION_THREAD_CO…

f638e40

…UNT annotations & comments

[controller] Refactor VeniceHelixAdmin::getStoresForCompaction logic …

2f3966d

…into CompactionManager - create CompactionManager & TestCompactionManager - move TestCompactionManager::testFilterStoresForCompaction from TestVeniceParentHelixAdmin

[controller] add scheduled.log.compaction.interval.HR config for exec…

8655b96

…utor setup in LogCompactionService::startInner

[controller][config] change schedule.log.compaction.interval.ms confi…

1f44e45

…g time unit from hour to millisecond

kvargha requested changes Feb 11, 2025

View reviewed changes

[pr] [style] config key naming camel case to lowercase

aa10e29

sixpluszero reviewed Feb 11, 2025

View reviewed changes

internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java Outdated Show resolved Hide resolved

Whitney Deng added 17 commits February 12, 2025 09:41

remove scheduled prefix

470b47c

[pr] [style] config key camel case to lowercase

8fac3bc

[pr] delete executionUrl

f2951ac

reason: not all repush orchestrators have execution UI url, but they all have execution ID

[logs][style] RepushJobRequest

5223a1c

- RepushJobRequest data model for repush job params - move triggerSource logging from LogCompactionTask to RepushJobRequest. reason: LogCompactionService is upstream from manual & scheduled common path - add source region as optional repush job param

[PR] remove Airflow configs

00c2273

[PR] remove Airflow configs

8997015

- delete RepushOrchestratorConfig, pass configs with VeniceProperties

[PR] remove Airflow in comment

2d6ecba

[PR][style] wildcard to single class imports

5e5fa20

[PR][style] wildcard to single class imports

6ce15e3

[PR][log] forceful shutdown log message for LogCompactionService exec…

b360da8

…utor

[PR][style] remove TODO

33c054e

[PR] throw error for missing RepushOrchestratorClassName config value

b9d0703

[PR] shut down log compaction service only if it is enabled

5551f4c

[PR][test] delete unused tests. LogCompactionService tests in TestHybrid

8b49945

[style] delete extra space

935c810

[PR][style][test] specify test time constant naming

fa37404

[PR][style][doc] return type documentation for Admin::compactStore

c004736

WhitneyDeng requested review from kvargha and lusong64 February 14, 2025 23:04

Whitney Deng added 4 commits February 14, 2025 15:26

[test] StatucAnalysis spotbugsIntegrationTest fix

e808d40

[test] testCompactionManager::testFilterStoresForCompaction() fix

77cbd7b

- isCompactionReady() needs to do real call

[test] remove test print statements

29376b8

[style] comments to explain method is public for testing

6cfd2b3

kvargha approved these changes Feb 18, 2025

View reviewed changes

lusong64 approved these changes Feb 18, 2025

View reviewed changes

WhitneyDeng merged commit b0babcf into linkedin:main Feb 18, 2025
58 checks passed

WhitneyDeng deleted the log-compaction branch February 18, 2025 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[controller] log compaction #1282

[controller] log compaction #1282

WhitneyDeng commented Nov 5, 2024

mynameborat left a comment

lusong64 left a comment

[controller] log compaction #1282

[controller] log compaction #1282

Conversation

WhitneyDeng commented Nov 5, 2024

Summary

How was this PR tested?

Does this PR introduce any user-facing changes?

mynameborat left a comment

Choose a reason for hiding this comment

lusong64 left a comment

Choose a reason for hiding this comment