Expose Platform under a feature flag

[rklaehn]

There are multiple projects that make use of some blake3 internals. Most notably bao, but also my bao-tree for n0/iroh and the fleek project.

There is a guts api, but that is pretty anemic. The unfortunate end result is that there are now several forks of blake3 on crates.io.

https://crates.io/crates/fleek-blake3
https://crates.io/crates/iroh-blake3

Until a better guts api is available, would it be an option to expose Platform under a feature flag? That would be a very low level guts API that requires a lot of knowledge about internals, but it would expose the entire SIMD optimized engine of blake3 to people that want to do something more complex. Basically something like blake3-sys.

[oconnor663]

I just wanted to let you know that I haven't forgotten about this! I have a lot of things working on my guts_api branch, and my short-term goal is to expose this as its own crate:

https://github.com/BLAKE3-team/BLAKE3/tree/guts_api/rust/guts

What I need to do is stabilize the API a bit and then release it with just the portable implementation. That'll let people start trying it out, without blocking on the large amount of work that needs to be done to port all the existing assembly code.

[rklaehn]

I just wanted to let you know that I haven't forgotten about this! I have a lot of things working on my guts_api branch, and my short-term goal is to expose this as its own crate:

https://github.com/BLAKE3-team/BLAKE3/tree/guts_api/rust/guts

What I need to do is stabilize the API a bit and then release it with just the portable implementation. That'll let people start trying it out, without blocking on the large amount of work that needs to be done to port all the existing assembly code.

Great. The fork works fine for now, but it is already stale, and it is causing all kinds of problems such as the C exports colliding when somebody uses the real blake3 crate. See n0-computer/iroh-blake3#2.

It would make me quite happy to get rid of it eventually.

The reason we are using the fork: iroh is using bao-tree with a chunk group size of 16 chunks. Without the fork, we don't get any parallelism when hashing chunk groups. With it - it is absolutely flying. Outboard creation is at > 1 GB/s on my machine. It is probably IO bound...

[devinrsmith]

Noting interest here too; there are some guts that I'd liked exposed that make streaming / bao-like consumption more possible.