Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(storage): store compaction group id in per table info #17125

Merged
merged 59 commits into from
Jun 13, 2024
Merged

feat(storage): store compaction group id in per table info #17125

Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
f587d0d
feat(storage): register member table id at commit epoch
wenym1 Mar 21, 2024
c802026
fix
wenym1 Mar 21, 2024
1216fcf
fix compile
wenym1 Mar 21, 2024
ffa06a4
Merge branch 'main' into yiming/register-hummock-table-in-commit
wenym1 Mar 21, 2024
c1188df
unregister table id in barrier command
wenym1 Mar 22, 2024
c091e59
fix compile
wenym1 Mar 22, 2024
5ea467e
feat(storage): support different snapshot for streaming jobs
wenym1 Mar 23, 2024
5d291f0
unregister member table id at commit_epoch
wenym1 Mar 23, 2024
b84b211
impl apply_version_delta
wenym1 Mar 23, 2024
80ee13e
fill backward compatibility snapshot group
wenym1 Mar 23, 2024
ad6fa20
Merge branch 'yiming/register-hummock-table-in-commit' into yiming/hu…
wenym1 Mar 23, 2024
f1b47b2
register snapshot group in commit epoch
wenym1 Mar 24, 2024
4025d64
impl purge
wenym1 Mar 25, 2024
0e50f63
fix
wenym1 Mar 25, 2024
e0b0697
fix compile
wenym1 Mar 25, 2024
a493db6
fix
wenym1 Mar 25, 2024
a03260f
Merge branch 'main' into yiming/register-hummock-table-in-commit
wenym1 Mar 25, 2024
f89ecd1
Merge branch 'yiming/register-hummock-table-in-commit' into yiming/hu…
wenym1 Mar 25, 2024
dbdd475
fix comment
wenym1 Mar 25, 2024
8318f5f
Merge branch 'yiming/register-hummock-table-in-commit' into yiming/hu…
wenym1 Mar 25, 2024
beb78a5
rename
wenym1 Mar 25, 2024
af38f68
fix compile
wenym1 Mar 25, 2024
778080b
impl system table
wenym1 Mar 25, 2024
5860e14
no extra info in create command
wenym1 Mar 25, 2024
f4dd843
unregister table id separately
wenym1 Mar 26, 2024
f934ba8
Merge branch 'yiming/register-hummock-table-in-commit' into yiming/hu…
wenym1 Mar 26, 2024
a8bf79d
refine
wenym1 Mar 26, 2024
e94774c
Merge branch 'main' into yiming/register-hummock-table-in-commit
wenym1 Mar 26, 2024
7cdca7b
Merge branch 'yiming/register-hummock-table-in-commit' into yiming/hu…
wenym1 Mar 26, 2024
2a41e01
add more panic log
wenym1 Mar 26, 2024
ef3af37
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Mar 29, 2024
6dbaebb
add prev id assertion
wenym1 Mar 29, 2024
b980553
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Mar 29, 2024
8000937
fix
wenym1 Mar 29, 2024
d53a32d
rename
wenym1 Mar 29, 2024
fd993d3
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Apr 8, 2024
fb87284
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Apr 10, 2024
28c50b5
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 May 7, 2024
307340d
Merge commit 'dac892e9777d4c249da3ae81fdf7b2d3d1218ae7' into yiming/h…
wenym1 May 21, 2024
11e7198
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 May 21, 2024
e57ed35
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 May 24, 2024
37df492
per table snapshot
wenym1 May 28, 2024
07d6d5d
feat(storage): store compaction group id in per table info
wenym1 May 31, 2024
ed2b4cc
extract state table info struct
wenym1 Jun 5, 2024
8735ebb
fmt
wenym1 Jun 5, 2024
0f31e14
Merge branch 'yiming/hummock-snapshot-group' into yiming/compaction-g…
wenym1 Jun 5, 2024
07ff5e2
extract rebuild index
wenym1 Jun 5, 2024
fc60078
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Jun 5, 2024
7c97d95
remove log
wenym1 Jun 5, 2024
b717e7c
fix compile
wenym1 Jun 5, 2024
c05e376
Merge branch 'main' into yiming/hummock-snapshot-group
wenym1 Jun 5, 2024
3ad89de
Merge branch 'yiming/hummock-snapshot-group' into yiming/compaction-g…
wenym1 Jun 5, 2024
8030edc
fix test
wenym1 Jun 5, 2024
a98bf38
Merge branch 'main' into yiming/compaction-group-in-table-info
wenym1 Jun 6, 2024
ef81602
Merge branch 'main' into yiming/compaction-group-in-table-info
wenym1 Jun 12, 2024
0ffc3ad
remove unnecessary logic
wenym1 Jun 12, 2024
0f55e99
Merge branch 'main' into yiming/compaction-group-in-table-info
wenym1 Jun 13, 2024
9e53e9a
fix comment
wenym1 Jun 13, 2024
ae1b046
Merge branch 'main' into yiming/compaction-group-in-table-info
wenym1 Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions proto/hummock.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,25 +69,28 @@ message IntraLevelDelta {
enum CompatibilityVersion {
VERSION_UNSPECIFIED = 0;
NO_TRIVIAL_SPLIT = 1;
NO_MEMBER_TABLE_IDS = 2;
}

message GroupConstruct {
CompactionConfig group_config = 1;
// If parent_group_id is not 0, it means parent_group_id splits into parent_group_id and this group, so this group is not empty initially.
uint64 parent_group_id = 2;
repeated uint32 table_ids = 3;
repeated uint32 table_ids = 3 [deprecated = true];
uint64 group_id = 4;
uint64 new_sst_start_id = 5;
CompatibilityVersion version = 6;
}

message GroupMetaChange {
repeated uint32 table_ids_add = 1;
repeated uint32 table_ids_remove = 2;
option deprecated = true;
repeated uint32 table_ids_add = 1 [deprecated = true];
repeated uint32 table_ids_remove = 2 [deprecated = true];
Comment on lines +86 to +88
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we can completely deprecate GroupMetaChange in GroupDelta?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because we won't update the member_table_ids anymore.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add deprecated in L107 then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added deprecated.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we miss [deprecated = true] in L107 for GroupMetaChange.

}

message GroupTableChange {
repeated uint32 table_ids = 1;
option deprecated = true;
repeated uint32 table_ids = 1 [deprecated = true];
uint64 target_group_id = 2;
uint64 origin_group_id = 3;
uint64 new_sst_start_id = 4;
Expand Down Expand Up @@ -152,11 +155,13 @@ message TableChangeLog {
message StateTableInfo {
uint64 committed_epoch = 1;
uint64 safe_epoch = 2;
uint64 compaction_group_id = 3;
}

message StateTableInfoDelta {
uint64 committed_epoch = 1;
uint64 safe_epoch = 2;
uint64 compaction_group_id = 3;
}

message HummockVersion {
Expand All @@ -165,7 +170,7 @@ message HummockVersion {
OverlappingLevel l0 = 2;
uint64 group_id = 3;
uint64 parent_group_id = 4;
repeated uint32 member_table_ids = 5;
repeated uint32 member_table_ids = 5 [deprecated = true];
}
uint64 id = 1;
// Levels of each compaction group
Expand Down
5 changes: 4 additions & 1 deletion src/meta/src/hummock/compaction/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

pub mod compaction_config;
mod overlap_strategy;
use risingwave_common::catalog::TableOption;
use risingwave_common::catalog::{TableId, TableOption};
use risingwave_pb::hummock::compact_task::{self, TaskType};

mod picker;
Expand Down Expand Up @@ -92,6 +92,7 @@ impl CompactStatus {
pub fn get_compact_task(
&mut self,
levels: &Levels,
member_table_ids: &HashSet<TableId>,
task_id: HummockCompactionTaskId,
group: &CompactionGroup,
stats: &mut LocalSelectorStatistic,
Expand All @@ -106,6 +107,7 @@ impl CompactStatus {
task_id,
group,
levels,
member_table_ids,
&mut self.level_handlers,
stats,
table_id_to_options.clone(),
Expand All @@ -121,6 +123,7 @@ impl CompactStatus {
task_id,
group,
levels,
member_table_ids,
&mut self.level_handlers,
stats,
table_id_to_options,
Expand Down
6 changes: 0 additions & 6 deletions src/meta/src/hummock/compaction/picker/base_level_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,6 @@ pub mod tests {
generate_table(1, 1, 201, 210, 1),
],
)],
member_table_ids: vec![1],
..Default::default()
};
let mut local_stats = LocalPickerStatistic::default();
Expand Down Expand Up @@ -418,7 +417,6 @@ pub mod tests {
total_file_size: 0,
uncompressed_file_size: 0,
}),
member_table_ids: vec![1],
..Default::default()
};
push_tables_level0_nonoverlapping(&mut levels, vec![generate_table(1, 1, 50, 140, 2)]);
Expand Down Expand Up @@ -481,7 +479,6 @@ pub mod tests {
total_file_size: 0,
uncompressed_file_size: 0,
}),
member_table_ids: vec![1],
..Default::default()
};
push_tables_level0_nonoverlapping(
Expand Down Expand Up @@ -585,7 +582,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(3, 1, 0, 100000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -663,7 +659,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(3, 1, 0, 100000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -730,7 +725,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(3, 1, 1, 100, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down
8 changes: 0 additions & 8 deletions src/meta/src/hummock/compaction/picker/intra_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -427,7 +427,6 @@ pub mod tests {
total_file_size: 0,
uncompressed_file_size: 0,
}),
member_table_ids: vec![1],
..Default::default()
};
push_tables_level0_nonoverlapping(
Expand Down Expand Up @@ -472,7 +471,6 @@ pub mod tests {
generate_table(1, 1, 100, 210, 2),
generate_table(2, 1, 200, 250, 2),
])),
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -510,7 +508,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -559,7 +556,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -631,7 +627,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
let mut levels_handler = vec![LevelHandler::new(0), LevelHandler::new(1)];
Expand Down Expand Up @@ -707,7 +702,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
levels_handler[1].add_pending_task(100, 1, levels.levels[0].get_table_infos());
Expand All @@ -728,7 +722,6 @@ pub mod tests {
let mut levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};
assert!(picker
Expand Down Expand Up @@ -826,7 +819,6 @@ pub mod tests {
let levels = Levels {
l0: Some(l0),
levels: vec![generate_level(1, vec![generate_table(100, 1, 0, 1000, 1)])],
member_table_ids: vec![1],
..Default::default()
};

Expand Down
4 changes: 4 additions & 0 deletions src/meta/src/hummock/compaction/picker/manual_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1198,6 +1198,7 @@ pub mod tests {
1,
&group_config,
&levels,
&HashSet::new(),
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -1235,6 +1236,7 @@ pub mod tests {
2,
&group_config,
&levels,
&HashSet::new(),
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -1308,6 +1310,7 @@ pub mod tests {
1,
&group_config,
&levels,
&HashSet::new(),
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -1347,6 +1350,7 @@ pub mod tests {
1,
&group_config,
&levels,
&HashSet::new(),
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down
28 changes: 22 additions & 6 deletions src/meta/src/hummock/compaction/picker/space_reclaim_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ mod test {
use std::sync::Arc;

use itertools::Itertools;
use risingwave_common::catalog::TableId;
use risingwave_pb::hummock::compact_task;
pub use risingwave_pb::hummock::{Level, LevelType};

Expand Down Expand Up @@ -234,11 +235,12 @@ mod test {
}

assert_eq!(levels.len(), 4);
let mut levels = Levels {
let levels = Levels {
levels,
l0: Some(l0),
..Default::default()
};
let mut member_table_ids = HashSet::new();
let mut levels_handler = (0..5).map(LevelHandler::new).collect_vec();
let mut local_stats = LocalSelectorStatistic::default();

Expand All @@ -252,6 +254,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand All @@ -268,6 +271,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -308,6 +312,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand All @@ -334,6 +339,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand All @@ -351,12 +357,17 @@ mod test {
}
}

levels.member_table_ids = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
member_table_ids = HashSet::from_iter(
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
.into_iter()
.map(TableId::new),
);
// pick space reclaim
let task = selector.pick_compaction(
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand All @@ -372,13 +383,15 @@ mod test {
}
}

levels.member_table_ids = vec![2, 3, 4, 5, 6, 7, 8, 9];
member_table_ids =
HashSet::from_iter([2, 3, 4, 5, 6, 7, 8, 9].into_iter().map(TableId::new));
// pick space reclaim
let task = selector
.pick_compaction(
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -407,7 +420,7 @@ mod test {
// rebuild selector
selector = SpaceReclaimCompactionSelector::default();
// cut range [3,4] [6] [8,9,10]
levels.member_table_ids = vec![0, 1, 2, 5, 7];
member_table_ids = HashSet::from_iter([0, 1, 2, 5, 7].into_iter().map(TableId::new));
let expect_task_file_count = [2, 1, 4];
let expect_task_sst_id_range = [vec![3, 4], vec![6], vec![8, 9, 10, 11]];
for (index, x) in expect_task_file_count.iter().enumerate() {
Expand All @@ -417,6 +430,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down Expand Up @@ -458,12 +472,13 @@ mod test {
// rebuild selector
selector = SpaceReclaimCompactionSelector::default();
// cut range [3,4] [6] [8,9,10]
levels.member_table_ids = vec![0, 1, 2, 5, 7];

member_table_ids = HashSet::from_iter([0, 1, 2, 5, 7].into_iter().map(TableId::new));
let expect_task_file_count = [2, 1, 5];
let expect_task_sst_id_range = [vec![3, 4], vec![6], vec![7, 8, 9, 10, 11]];
for (index, x) in expect_task_file_count.iter().enumerate() {
if index == expect_task_file_count.len() - 1 {
levels.member_table_ids = vec![2, 5];
member_table_ids = HashSet::from_iter([2, 5].into_iter().map(TableId::new));
}

// // pick space reclaim
Expand All @@ -472,6 +487,7 @@ mod test {
1,
&group_config,
&levels,
&member_table_ids,
&mut levels_handler,
&mut local_stats,
HashMap::default(),
Expand Down
1 change: 0 additions & 1 deletion src/meta/src/hummock/compaction/picker/tier_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,6 @@ pub mod tests {
sub_levels: vec![l1, l2],
}),
levels: vec![],
member_table_ids: vec![1],
..Default::default()
};
let config = Arc::new(
Expand Down
1 change: 0 additions & 1 deletion src/meta/src/hummock/compaction/picker/tombstone_reclaim_compaction_picker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,6 @@ pub mod tests {
],
),
],
member_table_ids: vec![1],
..Default::default()
};
let levels_handler = vec![
Expand Down
Loading