kafka replay speed: add alert for when we miss records in Kafka #9921
+122
−11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pracucci
approved these changes
Nov 15, 2024
Base automatically changed from
dimitar/ingest/remove-fetchWant-trimming
to
main
November 17, 2024 18:14
I realized that trimming `fetchWant`s can end up discarding offsets in extreme circumstances. ### How it works If the fetchWant is so big that its size would exceed 2GiB, then we trim it. We trim it by reducing the end offset. The idea is that the next fetchWant will pick up from where this one left off. ### How it can break We trim the `fetchWant` in `UpdateBytesPerRecord` too. `UpdateBytesPerRecord` can be invoked in `concurrentFEtchers.run` after the `fetchWant` is dispatched. In that case the next `fetchWant` would have already been calculated. And we would end up with a gap. ### Did it break? It's hard to tell, but it's very unlikely. To reach 2GiB we would have needed to have the estimation for bytes per record be 2 MiB. While these large records are possible, they should be rare and our rolling average estimation for records size shouldn't reach it. Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
dimitarvdimitrov
force-pushed
the
dimitar/ingest/detect-gaps-when-consuming
branch
from
aa30f73
to
90aaeeb
Compare
Signed-off-by: Dimitar Dimitrov <[email protected]>
tacole02
approved these changes
Nov 18, 2024
|
|
||
| How it **works**: | ||
|
|
||
| - Ingester reads records from Kafka, and processes them sequentially. It keeps track of the offset of the last record it processed. |
There was a problem hiding this comment.
Suggested change
| - Ingester reads records from Kafka, and processes them sequentially. It keeps track of the offset of the last record it processed. | |
| - The ingester reads records from Kafka and processes them sequentially. It keeps track of the offset of the last record it's processed. |
| How it **works**: | ||
|
|
||
| - Ingester reads records from Kafka, and processes them sequentially. It keeps track of the offset of the last record it processed. | ||
| - Upon fetching the next batch of records, it checks if the first available record has an offset one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. |
There was a problem hiding this comment.
Suggested change
| - Upon fetching the next batch of records, it checks if the first available record has an offset one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. | |
| - Upon fetching the next batch of records, it checks if the first available record has an offset of one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. |
|
|
||
| - Ingester reads records from Kafka, and processes them sequentially. It keeps track of the offset of the last record it processed. | ||
| - Upon fetching the next batch of records, it checks if the first available record has an offset one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. | ||
| - Kafka doesn't guarantee sequential offsets. If a record has been manually deleted from Kafka or the records have been produced in a transaction and the transaction was aborted, then there may be a gap. |
There was a problem hiding this comment.
Suggested change
| - Kafka doesn't guarantee sequential offsets. If a record has been manually deleted from Kafka or the records have been produced in a transaction and the transaction was aborted, then there may be a gap. | |
| - Kafka doesn't guarantee sequential offsets. If a record has been manually deleted from Kafka or if the records have been produced in a transaction and the transaction was aborted, then there may be a gap. |
| - Upon fetching the next batch of records, it checks if the first available record has an offset one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. | ||
| - Kafka doesn't guarantee sequential offsets. If a record has been manually deleted from Kafka or the records have been produced in a transaction and the transaction was aborted, then there may be a gap. | ||
| - Mimir doesn't produce in transactions and does not delete records. | ||
| - When the ingester starts up, it will attempt to resume from the last offset it processed. If the ingester has been unavailable for long enough that the next record is already removed due to retention, then the ingester will miss some records. |
There was a problem hiding this comment.
Suggested change
| - When the ingester starts up, it will attempt to resume from the last offset it processed. If the ingester has been unavailable for long enough that the next record is already removed due to retention, then the ingester will miss some records. | |
| - When the ingester starts, it attempts to resume from the last offset it processed. If the ingester has been unavailable for long enough that the next record is already removed due to retention, then the ingester misses some records. |
There was a problem hiding this comment.
We avoid using future tense in the docs.
| - Ingester reads records from Kafka, and processes them sequentially. It keeps track of the offset of the last record it processed. | ||
| - Upon fetching the next batch of records, it checks if the first available record has an offset one greater than the last processed offset. If the first available offset is larger than that, then the ingester has missed some records. | ||
| - Kafka doesn't guarantee sequential offsets. If a record has been manually deleted from Kafka or the records have been produced in a transaction and the transaction was aborted, then there may be a gap. | ||
| - Mimir doesn't produce in transactions and does not delete records. |
There was a problem hiding this comment.
"Mimir doesn't produce in transactions" reads unclear to me. Is the "in" supposed to be here?
|
|
||
| How to **investigate**: | ||
|
|
||
| - Verify that there have been no deleted records in your Kafka cluster by humans or other applications. |
There was a problem hiding this comment.
I think we can probably remove "by humans or other applications".
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds an alert and metrics to detect when we have bugs.
Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.