FR: allow schema evolution (in or after 0.6)

[forrestli74]

I know it would be serving really different audience if it supports schema evolution (SE for short). But I would really benefit from similar encoding as bitcode but with SE and it would be a lot of duplicate code to implement such thing. So I'm here to ask: will you consider adding SE as a flavor of bitcode? Or somehow allow me to write a crate that easily reuses bitcode encoding and add SE?

I think the main thing that schema evolution need is

1. appending fields to struct (the encoding will store the number of fields in struct and new fields will be deserialized to default)
2. appending variant to enum
3. increase capacity of ints (I.e. u8 to u16) (though this is less important)

In 0.6 style encoding, this would not add much space either.

[caibear]

We considered schema evolution and self description when we made 0.6, but we came to the conclusion that those features require a stable format to be useful. We currently have a lot of freedom when it comes to future optimizations because we're able to change the format each major version.

I do agree that schema evolution is useful, however I think it's outside the scope of bitcode. Funnily enough as you sent this I'm currently experimenting with self describing/schema evolution via serde that would work with any binary serialization including bitcode. Hopefully I get somewhere with it. The benefit to putting it outside bitcode is that you could keep your data across major versions of bitcode.

[forrestli74]

Make sense. Serde might be able to help with schema evolution on top of bitcode. Would be interesting to see how it works. Feel free to close.
edit follow up: will bitcode ever have a stable format? 🤣

[caibear]

edit follow up: will bitcode ever have a stable format? 🤣

probably not, unless I stop working on it 🤣