Build script: Adding directive cargo::no-new-rust-code to speed up compilation

[NobodyXu]

Problem

Currently, all crates that bindgens, compiles or finds external dependencies have to wait for compilation/lookup of compiler artifacts to be done before compilation can continue.

For example, zstd-sys takes really a long time to compile zstd and all crates that depend on it have to wait until they are compiled.

Given that bindgen usually doesn't take that long (if user enabled zstd-sys/bindgen), the dependents of zstd-sys could start compilation earlier.

Proposed Solution

I propose that we introduce a new flag cargo::no-new-rust-code-change, it would tell cargo that the build script won't do anything that affects compilation of the rust code of this crate (generating code, issuing cfgs, etc), and so the rust crate can generate the rmeta and compile the code.

Once the rmeta is generated, dependents of this crate can start compiling without waiting for the whole compilation to be done.

If the build-script violates the protocol and actually tries to issue build-script instructions for cfgs then compilation would fail.

Generating code or changing already generated code that the rust crate includes!() would be considered the same as trying to modify the source code of the crate when rustc is compiling, rustc/cargo does not guarantee whether it would be read, and rustc could read the files when build script is updating it.

implementation

Implementing this would need -Zno-link to defer archiving to -Zlink-only so that it doesn't rely on external static libraries produced by the build script.

Notes

how would that benefit systems with less than 4 cores, e.g. GHA?

It definitely cannot benefit single-core system, however for system with 2-4 cores it could still benefit.

build-script using cc-rs

P.S. I am one of the maintainers of cc-rs

For crate using cc-rs to compile the code, cc-rs cannot fully utilise every core at all time.

Particularly, cc-rs runs is_flag_supported logic in sequential, and that logic actually tries to compile a dummy c file to check if the compiler supports the flag.

cc-rs also has other places where it has to run external command sequentially to retrieve information (checking compiler family, looking for macOS/windows env, etc).

During parallel compilation (if feature cc/parallel is enabled), cc is not guaranteed to spawn a new job immediately when a job token is freed by another process (e.g. other build scripts, rustc, etc).

compilation using external systems

Compilation using external systems such as configure + make, or cmake (potentially through cmake-rs) typically involves a setup/env detection and makefile/ninja generation process that AFAIK is single thread and takes quite some time.

It might be enough for rustc to compile the crate itself and unblock the dependents to be compiled in parallel.

And since ninja/make has an internal dependencies resolution process, it is not likely to fully utilise all cores at all time.

[epage]

This was also discussed on Internals

[epage]

This seems similar in spirit to pipelined builds we do for Rust code where we allow downstream dependencies to build once the rmeta is created, only requiring the rlib at link time.

I suspect the first step in doing this would be to redefine build scripts as the directives being processed on the fly, rather than at build script completion. Support for doing so was a consideration in the discussions for #13233.

Unsure what all it would take architecturally in Cargo to have build script execution participate in the pipelined builds setup.

Potential routes forward:

• We consider all of the ins and outs of this and make a decision and move forward
• We identify small pieces we'd be willing to accept independent of this (e.g. processing build scripts on the fly) and then work towards a prototype to see what we think

Unsure which route we should go and how much we'd be willing to accept as part of this.

[NobodyXu]

Not familiar with how cargo internally works, it seems to spawn one thread per rustc/build-script call to track the tasks?

In cc-rs we used to do that, but now we switched to using async to do spawning waiting for children, so that we don't have to spawn two threads for each compiler invocation:

https://github.com/rust-lang/cc-rs/blob/a14c5181cd411f6d4a562e4017a6a4ffdb1196ea/src/parallel/command_runner.rs#L99

And use a super dumb async reactor becaue we don't want to pull in extra crate or maintain it

https://github.com/rust-lang/cc-rs/blob/a14c5181cd411f6d4a562e4017a6a4ffdb1196ea/src/parallel/async_executor.rs#L93

Since cargo doesn't have the constraint cc-rs have, maybe it can just go full async, and use async to keep track of states?

[kornelski]

I could make use of that in sys crates that run bindgen + compile fallback from source.

Splitting of build across multiple scripts doesn't seem helpful to me.

The problem is that bindgen must be run on the right version of the header files, which first requires detecting whether the sys library has a pre-installed version or needs to use a vendored one. If I had to split the steps into two build scripts, I would have to do this detection in each build script, duplicating the work (with a risk of two halves disagreeing if the detection goes out of sync for some reason). If there was some dependency graph and communication between multiple split build scripts I could make detection a one build script and then bindgen and compile from sources its children, but that starts looking overly complicated, and linking and potential overzealous antivirus checks of three build script executables will probably make everything slower.

---

BTW: this directive probably also means no new cargo::metadata directives can be used, since the env vars they create are visible to Rust code in other deps.

[zachs18]

Semi-related rambling about possible usage of RFC 3632 "externally implementable functions" in build scripts

Maybe we could introduce "Build Script V2 ™️ " using something like rust-lang/rfcs#3632 , and have different "stages" of the build script as different functions the user can implement, e.g. something like

impl fn build_script_v2::bikeshed_before_rust_build() {
  // stuff that must complete before `rustc` can begin building this crate's source, e.g. code generation
  // or emitting directives that change cfgs
}

impl fn build_script_v2::bikeshed_concurrent_with_rust_build() {
  // stuff that can happen concurrent with `rustc` producing `.rmeta`, but before linking,
  // like running external build systems for `-sys` crates
}

// etc if there are other stages; all have a default body that does nothing

A "Build Script V2 ™️ " build script would be a library crate-type (not a binary itself) and would be compiled in as a dependency of a simple binary (source distributed with cargo and compiled on-the-fly) that calls the functions in the relevant order, telling cargo when each one has completed. If a directive is emitted in the "wrong" stage (i.e. after it is allowed), cargo would detect the error and fail compilation of the crate.

Also, perhaps some stages could be only executed during a cargo build, not during a cargo check.

[NobodyXu]

Maybe we could introduce "Build Script V2 ™️ " using something like rust-lang/rfcs#3632

That's interesting idea, if cargo build scripts is redesigned today then it'd surely be like this, plus combined with declarative build scripts, though I think it's orthogonal to this proposal, as it is a more radical and fundamental change?

I personally want to pursue a more incremental change that can be applied to existing build scripts relatively easily.

You can open a separate issue for that.