Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425

[rami3l]

@justsmth thank you so much for your investigation and hard work so far! I'll see you again very soon in other follow-up issues 😶
#453 (comment)

Here we go 🤦‍♀️

Problem

After merging rust-lang/rustup#3898, Rustup's FreeBSD CI has started failing.

rust-lang/rustup#3979 tries to set up the necessary dependencies for the build, namely cmake-core and llvm-devel-lite (for libclang.so). Since it's internal bindgen by default, bindgen-cli doesn't seem to be required.

But still, Compiling aws-lc-sys v0.20.1 seems to have failed with the following log:

  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:74:14
     |
  74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:80:14
     |
  80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_FUNC_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:86:14
     |
  86 |     unsafe { ERR_GET_FUNC_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `BIO_ctrl` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:91:14
     |
  91 |     unsafe { BIO_ctrl(b, BIO_CTRL_INFO, 0, pp.cast::<c_void>()) }
     |              ^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `CRYPTO_library_init` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:95:14
     |
  95 |     unsafe { CRYPTO_library_init() }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
  For more information about this error, try `rustc --explain E0425`.
  The following warnings were emitted during compilation:
  warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
  warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
  error: could not compile `aws-lc-sys` (lib) due to 5 previous errors

https://github.com/rust-lang/rustup/actions/runs/10192459718/job/28195406529?pr=3979

Relevant details

FWIW, we use the following action for our FreeBSD tests:

- uses: vmactions/freebsd-vm@v1
  with:
    release: 13.2
    usesh: true
    copyback: false
    prepare: |
      pkg install -y git gmake bash sudo cmake-core llvm-devel-lite

https://github.com/rust-lang/rustup/blob/8fdc61316891cdb72acf406b2ac929227ceb0068/ci/actions-templates/freebsd-builds-template.yaml

[justsmth]

Hello again! 😊

I've seen this type of failure before. It typically occurs after the bindings are generated, but the logic in lib.rs fails to pick it up. (Perhaps this include! needs an adustment?) I'll dig in now.

[justsmth]

Some progress made. I'm able to reproduce the same failure in the FreeBSD CI test I'm setting up: https://github.com/aws/aws-lc-rs/actions/runs/10198121880/job/28212416541#step:3:38543

...
      Compiling aws-lc-rs v1.8.1 (/home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-rs)
     Compiling clap_derive v4.5.13
     Compiling clap_builder v4.5.13
  warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
  warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
    --> aws-lc-sys/src/lib.rs:74:14
     |
  74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^ not found in this scope
  
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
    --> aws-lc-sys/src/lib.rs:80:14
     |
  80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
...

I'm working on setting up a dedicated FreeBSD host so I can debug more easily.

[justsmth]

More progress... I can also reproduce on a FreeBSD EC2 instance:

   Compiling paste v1.0.15
   Compiling aws-lc-sys v0.20.1 (/usr/home/ec2-user/repos/aws-lc-rs/aws-lc-sys)
warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
  --> aws-lc-sys/src/lib.rs:74:14
   |
74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
   |              ^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
  --> aws-lc-sys/src/lib.rs:80:14
   |
80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
   |              ^^^^^^^^^^^^^^^^^^^ not found in this scope

Now to the "fun" part... 😊

[justsmth]

So it appears that the "internal" bindings generation is producing invalid bindings, however using the "external" bindgen-cli works fine and all tests pass:

❯ AWS_LC_SYS_EXTERNAL_BINDGEN=1 cargo test -p aws-lc-rs
  Downloaded anstyle-parse v0.2.5
  Downloaded colorchoice v1.0.2
...
warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
warning: aws-lc-sys@0.20.1: Generating bindings - external bindgen. Platform: x86_64-unknown-freebsd
    Finished `test` profile [unoptimized + debuginfo] target(s) in 52.02s
     Running unittests src/lib.rs (target/debug/deps/aws_lc_rs-dcf25f61ef54de42)

running 187 tests
...
test rsa4096_oaep_sha256_mgf1sha256 ... ok
test rsa4096_oaep_sha512_mgf1sha512 ... ok

test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.95s

   Doc-tests aws_lc_rs

running 24 tests
test aws-lc-rs/src/cipher.rs - cipher (line 140) ... ok
...
test aws-lc-rs/src/tls_prf.rs - tls_prf (line 8) ... ok
test aws-lc-rs/src/pbkdf2.rs - pbkdf2 (line 23) ... ok

test result: ok. 22 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 1.61s

So a temporary workaround might be to ensure that "bindgen-cli" is installed and have AWS_LC_SYS_EXTERNAL_BINDGEN=1 set in the environment. I'll try this for the PR I'm working on for this.

I'll keep digging to see why the internal bindgen is behaving like this.

---

Edit: The FreeBSD CI test succeeded using external bindgen: https://github.com/aws/aws-lc-rs/actions/runs/10199677822/job/28217403628#step:3:39141

[rami3l]

So a temporary workaround might be to ensure that "bindgen-cli" is installed and have AWS_LC_SYS_EXTERNAL_BINDGEN=1 set in the environment.

@justsmth Thanks a lot for the pointers! It seems to work with Rustup as well.

Also, adding rust-bindgen-cli to the pkg install -y line will save you a minute or two from using cargo install.

I'll keep digging to see why the internal bindgen is behaving like this.

Please take your time! It's not really a blocker for us now that we have this workaround :)

[keehun-cf]

I've noticed this exact error message when building on Windows 11 using clang/LLVM 19.1.0. As soon as I downgraded to 18.1.8, my builds just started working again.

I'd be happy to create a minimal reproduction case if anyone is interested.

[justsmth]

I'd be happy to create a minimal reproduction case if anyone is interested.

@keehun-cf - Yes, that would be great!

I've noticed this exact error message when building on Windows 11 using clang/LLVM 19.1.0. As soon as I downgraded to 18.1.8, my builds just started working again.

This is a very interesting data point. I'm curious whether you were using our (internal) "bindgen" feature when the build failed, or having the build use the (external) bindgen-cli? This can be forced either way by setting AWS_LC_SYS_EXTERNAL_BINDGEN=1 in the build environment.

Speculating... perhaps the bindgen-cli (latest version, v0.70.1) can work properly with clang/LLVM 19.1.0, while the bindgen version we depend on (v0.69.4) can not?

One concern I've had is that we're stuck at bindgen v0.69 due to bindgen v0.70 bumping the MSRV to 1.70. (Our MSRV is 1.63).