backfill: Impossible to implemented custom store #638
Comments
|
If you have small PRs, we might be able to review and merge them, but no promises. Right now it could take us a few weeks. It helps if you can keep them distinct, eg operation contstants separate from ops. |
This was referenced Apr 9, 2024
ericvolp12
added a commit
that referenced
this issue
Apr 16, 2024
(Part of #638) This utilizes constants from the `repomgr` package for buffered operations. This was already the case, actually, as they were used [here](https://github.com/bluesky-social/indigo/blob/main/backfill/backfill.go#L241), I just added them throughout the code. They're also used to check the operations emitted by the firehose: https://github.com/bluesky-social/indigo/blob/main/backfill/backfill.go#L469-L471 The firehose, in turn, uses the events emitted by the repo manager for the `Action` field in firehose events: https://github.com/bluesky-social/indigo/blob/main/indexer/indexer.go#L83-L94 This brings us full circle: Repo manager emits diff operations -> firehose repo commit ops `.Action` -> backfill buffered op `.kind` Notably, the constants are also given in the lexicon: https://github.com/bluesky-social/atproto/blob/main/lexicons/com/atproto/sync/subscribeRepos.json#L178 But `lexgen` does not generate constants for them. The test in `backfill_test.go` is broken, but I changed the constants anyway.
ericvolp12
added a commit
that referenced
this issue
May 2, 2024
(part of #638) This exports the `backfill.BufferedOp` struct to make it possible to implement a custom `Store` and corresponding `Job`. I've also renamed the fields slightly for easier understanding, and added some documentation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To work around #629, I tried creating a custom
backfill.Store. This, however, is not possible because thebackfill.Jobinterface depends on the unexported structbackfill.bufferedOpvia theBufferOpsmethod:https://github.com/bluesky-social/indigo/blob/main/backfill/backfill.go#L33
(Nitpick: it would also be helpful to have the operation kinds (
"create", etc.) as constants.)Should I open a PR for these?
The text was updated successfully, but these errors were encountered: