iroh doesn't compiles on Android with Neon - ld: error: duplicate symbol: blake3_compress_in_place_portable

[Voronar]

Android build settings:

android {
    compileSdk 34
    ndkVersion "25.2.9519653"

    defaultConfig {
        minSdk 24
        targetSdk 33
    }
}

iroh: v0.10

Error:

ld: error: duplicate symbol: blake3_compress_in_place_portable
          >>> defined at ffi_neon.rs:37 (src/ffi_neon.rs:37)
          >>>            iroh_blake3-1c414b619f1f58b4.iroh_blake3.ac6b13366874c9bc-cgu.0.rcgu.o:(blake3_compress_in_place_portable) in archive

But if I disable Neon in my app crate with iroh-blake3 = { version = "*", features = ["no_neon"]} it compiles.

[dignifiedquire]

n0-computer/iroh-blake3#2 should fix this (hopefully)

[rklaehn]

So the issue is that we had to fork blake3 to get some performance improvements for bao in. But it exports symbols, so if you have the normal blake3 in your dependencies there will be a collision.

One way to solve this would be to directly use our iroh-blake3 instead of blake3 in your other code. But we realize that this is not always possible.

We hope to get these very minor changes upstreamed to blake3 soon.

[link2xt]

Same here, just failed to compile deltachat/deltachat-core-rust#5041 on Android.
We have blake3 from iroh 0.4 used for backup transfer and iroh-blake3 from recent iroh-gossip/iroh-net etc.

starting time: Fri Nov 24 10:30:45 UTC 2023
Setting CARGO_TARGET environment variables.
Full build
-- cross compiling to aarch64-linux-android (arm64) --
   Compiling deltachat_ffi v1.131.6 (/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/deltachat-ffi)
warning: Linking globals named 'blake3_compress_in_place_portable': symbol multiply defined!

error: failed to load bitcode of module "iroh_blake3-5d1abe52b19e03f8.iroh_blake3.d776915badce0b10-cgu.0.rcgu.o": 

warning: `deltachat_ffi` (lib) generated 1 warning
error: could not compile `deltachat_ffi` (lib) due to previous error; 1 warning emitted

I tried to use

[replace]
"blake3:1.5.0" = { package = "iroh-blake3" }

But it tries to replace with iroh-blake3:1.5.0 which does not exist, and I cannot specify version = 1.4.3 (this is what we have in Cargo.lock for iroh-blake3 because of rust-lang/cargo#2649

Trying to also use patch for replacing blake3 version fails with

error: failed to parse manifest at `/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/Cargo.toml`

Caused by:
  cannot specify both [replace] and [patch]

If I try to use only patch, I fail because of rust-lang/cargo#9227:

[patch.crates-io]
blake3 = { version = "1.4.3", package = "iroh-blake3" }

results in

error: failed to resolve patches for `https://github.com/rust-lang/crates.io-index`

Caused by:
  patch for `iroh-blake3` in `https://github.com/rust-lang/crates.io-index` points to the same source, but patches must point to different sources

[link2xt]

I also looked into fixing it in a generic way (with -fvisibility=hidden or so), but there is a single linking step that links libdeltachat, libblake3 and libiroh-blake3 together, at which point libdeltachat cannot say whether it wants a C symbol from libblake3 or libiroh-blake3 at each place:

CARGO_PKG_VERSION_MAJOR=1
CARGO_PKG_VERSION_MINOR=131
CARGO_PKG_VERSION_PATCH=6
CARGO_PKG_VERSION_PRE=''
CARGO_PRIMARY_PACKAGE=1
LD_LIBRARY_PATH='/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/release/deps:/nix/store/8d4hd5s79qp78jyppikcm2m354jd3zk1-rust-minimal-1.72.0/lib'
OUT_DIR=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/deltachat_ffi-47af51b513a377a2/out
RUST_MIN_STACK=8388608
rustc
--crate-name deltachat
--edition=2018 deltachat-ffi/src/lib.rs
--error-format=json 
--json=diagnostic-rendered-ansi,artifacts,future-incompat
--diagnostic-width=191
--crate-type cdylib
--crate-type staticlib
--emit=dep-info,link -C opt-level=z -C panic=abort -C lto=on -C codegen-units=1
--cfg 'feature="default"' --cfg 'feature="jsonrpc"' --cfg 'feature="vendored"'
-C metadata=de605aeb2b0c3a1c --out-dir /home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps
--target aarch64-linux-android
-C linker=/nix/store/7liincj7f4sqd89zhihjpsvjbqlmhpdv-android-sdk-env/share/android-sdk/ndk/23.2.8568313/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android21-clang
-C strip=symbols
-L dependency=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps
-L dependency=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/release/deps
--extern anyhow=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libanyhow-a14a390d0d20a18e.rlib
--extern deltachat=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libdeltachat-b6a2bd6038eeb4ec.rlib
--extern deltachat_jsonrpc=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libdeltachat_jsonrpc-78475ddd78d6da44.rlib
--extern human_panic=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libhuman_panic-06ae38872aa4e2b0.rlib
--extern libc=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/liblibc-46971386cb7c3e56.rlib
--extern num_traits=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libnum_traits-277cdcbd8cb8912b.rlib
--extern once_cell=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libonce_cell-67aee0e2aab68aed.rlib
--extern rand=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/librand-aeb0202ba3689b17.rlib
--extern serde_json=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libserde_json-caf71276b8e61d01.rlib
--extern thiserror=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libthiserror-685cd9c0a7f318cc.rlib
--extern tokio=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libtokio-8f13811cabeb1ec9.rlib
--extern yerpc=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/deps/libyerpc-902fcce38fa84673.rlib
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/openssl-sys-4cf047d6a8426c95/out/openssl-build/install/lib
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/blake3-a56da7ee5bc19a3c/out
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/ring-b0a8a65fbc35d9eb/out
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/ring-38dc824764cbf6e7/out
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/iroh-blake3-b1cab4eb62a607cd/out
-L native=/home/user/src/deltachat/deltachat-android/jni/deltachat-core-rust/target/aarch64-linux-android/release/build/libsqlite3-sys-365ce9e994955565/out`

[flub]

@link2xt did you see n0-computer/iroh-blake3#2 as the fix to this?

We should merge and depend on that really.

[link2xt]

@flub Yes, it is linked above but seems to be failing CI and somewhat stale.

Another option would be to depend on the latest iroh-blake3 in #1821

[link2xt]

Now that n0-computer/iroh-blake3#2 is merged, I can do

[patch.crates-io]
"iroh-blake3" = { git = "https://github.com/n0-computer/iroh-blake3.git", branch = "master" }

and this compiles on Android.

I compiled deltachat/deltachat-core-rust#5041 and gossip works, but for some reason sending works only from Desktop to Android but not from Android to Desktop. Probably has something to do with missing syscalls on old Linux kernel that my Android phone is running and stuff like that, at least now I can debug it.

[rklaehn]

i think this should be solved by the changes in iroh-blake3, and will be solved permanently by the new blake3 guts api that will allow us to directly build on blake3 without fork: https://github.com/BLAKE3-team/BLAKE3/tree/guts_api/rust/guts

[dignifiedquire]

I believe this is fixed in the latest releases