Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(meta): reduce time travel hummock version size #18699

Merged
merged 1 commit into from
Sep 27, 2024
+11 −2
Merged

refactor(meta): reduce time travel hummock version size #18699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/meta/src/hummock/manager/time_travel.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,7 @@ impl HummockManager {
.order_by_asc(hummock_time_travel_delta::Column::VersionId)
.all(&sql_store.conn)
.await?;
// SstableInfo in actual_version is incomplete before refill_version.
let mut actual_version = replay_archive(
replay_version.version.to_protobuf(),
deltas.into_iter().map(|d| d.version_delta.to_protobuf()),
Expand Down Expand Up @@ -363,7 +364,7 @@ impl HummockManager {
query_epoch, actual_version_id,
))));
}
refill_version(&mut actual_version, &sst_id_to_info);
refill_version(&mut actual_version, &sst_id_to_info, table_id);
Ok(actual_version)
}

Expand Down
10 changes: 9 additions & 1 deletion src/storage/hummock_sdk/src/time_travel.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,12 @@ use crate::{CompactionGroupId, HummockSstableId};

pub type IncompleteHummockVersion = HummockVersionCommon<SstableIdInVersion>;

/// Populates `SstableInfo` for `table_id`.
/// `SstableInfo` not associated with `table_id` is removed.
pub fn refill_version(
version: &mut HummockVersion,
sst_id_to_info: &HashMap<HummockSstableId, SstableInfo>,
table_id: u32,
) {
for level in version.levels.values_mut().flat_map(|level| {
level
Expand All @@ -41,8 +44,13 @@ pub fn refill_version(
.chain(level.levels.iter_mut())
}) {
refill_level(level, sst_id_to_info);
level
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we apply the table_id filter when we refill_level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to get the true SstableInfo from HashMap<HummockSstableId, SstableInfo> to tell whether the sst can be omitted. So filtering out by table_id in inner calls like refill_level can only enhance performance by a assignment of SstableInfo, which I think is not significant.

.table_infos
.retain(|t| t.table_ids.contains(&table_id));
}

version
.table_change_log
.retain(|t, _| t.table_id == table_id);
for t in version.table_change_log.values_mut() {
refill_table_change_log(t, sst_id_to_info);
}
Expand Down
Loading