refactor: add target_offsets to determinine if source backfill finished #18297
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Base automatically changed from
08-28-refactor_simplify_sourcebackfill_s_memory_state
to
main
August 28, 2024 09:32
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
2 times, most recently
from
e9f69ef
to
27cf1fc
Compare
xxchan
commented
Aug 30, 2024
| @@ -727,6 +766,18 @@ impl<S: StateStore> SourceBackfillExecutorInner<S> { | |||
| debug_assert_eq!(old_states, target_state); | |||
| } | |||
| stage.states = target_state; | |||
| stage.splits = target_splits; | |||
There was a problem hiding this comment.
This line fixed a bug https://github.com/risingwavelabs/risingwave/pull/18296/files#r1738292017
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
from
27cf1fc
to
aa5610d
Compare
9 tasks
xxchan
requested review from
chenzl25,
fuyufjh,
tabVersion,
BugenZhao and
st1page
BugenZhao
approved these changes
Sep 2, 2024
Comment on lines
+770
to
+780
| let old_target_offsets = std::mem::take(&mut stage.target_offsets); | ||
| stage.target_offsets = stage | ||
| .states | ||
| .keys() | ||
| .map(|split_id| { | ||
| ( | ||
| split_id.clone(), | ||
| old_target_offsets.get(split_id).cloned().flatten(), | ||
| ) | ||
| }) | ||
| .collect(); |
There was a problem hiding this comment.
Need to consider newly added splits.
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
from
aa5610d
to
272a7a4
Compare
9 tasks
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
from
c8b80b7
to
2b8f062
Compare
9 tasks
xxchan
deleted the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
This was referenced Sep 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
step 1 of #18299, see there for the motivation and a state diagram of the new design.
The main idea of the code change is to consolidate logic into
handle_upstream_rowandhandle_backfill_row.Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.