Skip to content

Latest commit

 

History

History
48 lines (34 loc) · 3.3 KB

README.md

File metadata and controls

48 lines (34 loc) · 3.3 KB
Raw

Margaret Go Reference Github Actions Tests Go Report Card REUSE status

Margaret is go-ssb's append-only log* provider, and greatly inspired by flumedb. Compatible with Go 1.13+.

margaret the log lady, 1989 edition

the project name is inspired by Twin Peaks's character Margaret aka the log lady

Margaret has the following facilities:

  • an append-only log interface .Append(interface{}), .Get(int64)
  • queries .Query(...QuerySpec) for retrieving ranges based on sequence numbers e.g. .Gt(int64), or limiting the amount of data returned .Limit(int64)
  • a variety of index mechanisms, both for categorizing log entries into buckets and for creating virtual logs (aka sublogs)

Margaret is one of a few key components that make the go implementation of ssb tick, for example:

  • ssb/sbot uses margaret for storing each peer's data

Log storage

Margaret outputs data according to the offset2 format, which is inspired by (but significantly differs from) flumelog-offset.

In brief: margaret stores the data of all logs in the three following files:

  • data stores the actual data (with a length-prefix before each entry)
  • ofst indexes the starting locations for each data entry in data
  • jrnl an integrity checking mechanism for all three files; a checksum of sorts, more details

More details

There are a few concepts that might be tough to digest for newcomers on first approach:

  • multilogs, a kind of tree-based index, where each leaf is a margaret.Log
    • in other words: it creates virtual sublogs that map to entries in an offset log (see log storage above)
  • margaret/indexes similar to leveldb indexes (arbitrary key-value stores)
  • sublogs (and rxLog/receiveLog/offsetLog and its equivalence to offset.log)
  • queries
  • zeroing out, or replacing, written data

For more on these concepts, visit the dev.scutttlebutt.nz portal for in-depth explanations.

* margaret is technically an append-based log, as there is support for both zeroing out and replacing items in the log after they have been written. Given the relative ubiquity of append-only logs & their uses, it's easier to just say append-only log.