Concurrent `cargo check` with a shared `target` directory is broken since `1.93.0-beta.1`

[wmmc88]

Problem

When you call cargo check concurrently on two packages and use the same target directory, the check fails because it looks like cargo concurrently writes to target directory. This does not happen with cargo build.

This seems to be a change introduced in nightly-2025-11-21, reverted a while later which fixed it, but then was later reintroduced at some point. I noticed this issue in 1.93.0-beta.1 as well as the latest 1.94.0-nightly (f52090008 2025-12-10).

Originally reported in eupn/macrotest#131, since macrotest relies on concurrent cargo expand calls on crates with a shared target directory.

Possibly related to #16297.

Steps

Since this repro requires running 2 concurrent cargo check calls with a shared target directory, I created a minimal repro repo w/ a script: https://github.com/wmmc88/minimal-repros/tree/rust-nightly-2025-11-21-bug

1. clone above minimal example
2. .\repro.ps1 -CargoCommand check -CargoToolchain 1.93.0-beta.1
3. Get following error output:

🪟 Windows 11 Enterprise(10.0.26200) (pwsh) minimal-repros on  rust-nightly-2025-11-21-bug [!] took 5s
at 04:52:40 PM ❯ .\repro.ps1 -CargoCommand check -CargoToolchain 1.93.0-beta.1
Set CARGO_TARGET_DIR to D:\git-repos\github\minimal-repros/shared-target
Cleaning CARGO_TARGET_DIR...
Starting builds...
Using cargo toolchain override: 1.93.0-beta.1                                                                           
Cargo command: cargo check                                                                                              
[P1] Running: cargo +1.93.0-beta.1 check --manifest-path D:\git-repos\github\minimal-repros\package-a\Cargo.toml        
[P1]    Compiling proc-macro2 v1.0.103                                                                                  
[P1]     Checking unicode-ident v1.0.22                                                                                 
[P1]    Compiling quote v1.0.42                                                                                         
[P1]     Checking syn v2.0.111                                                                                          
[P1]     Checking package-a v0.1.0 (D:\git-repos\github\minimal-repros\package-a)                                       
[P1]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.24s                                            
[P1] Exit code: 0                                                                                                       
[P2] Running: cargo +1.93.0-beta.1 check --manifest-path D:\git-repos\github\minimal-repros\package-b\Cargo.toml        
[P2]    Compiling proc-macro2 v1.0.103                                                                                  
[P2]    Compiling quote v1.0.42
[P2] error: linking with `link.exe` failed: exit code: 1104                                                             
[P2]   |                                                                                                                
[P2]   = note: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64\\link.exe" "/NOLOGO" "C:\\Users\\MELVIN~1\\AppData\\Local\\Temp\\rustcIjhHki\\symbols.o" "<3 object files omitted>" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib/{libstd-*,libpanic_unwind-*,libcfg_if-*,libwindows_targets-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libunwind-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "kernel32.lib" "kernel32.lib" "kernel32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:\\git-repos\\github\\minimal-repros/shared-target\\debug\\build\\proc-macro2-2646f7cb26a09dc3\\build_script_build-2646f7cb26a09dc3.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libstd.natvis"
[P2]   = note: some arguments are omitted. use `--verbose` to show all linker arguments                                 
[P2]   = note: LINK : fatal error LNK1104: cannot open file 'D:\git-repos\github\minimal-repros\shared-target\debug\build\proc-macro2-2646f7cb26a09dc3\build_script_build-2646f7cb26a09dc3.exe'ΓÉì
[P2]                                                                                                                    
[P2] System.Management.Automation.RemoteException                                                                       
[P2] error: could not compile `proc-macro2` (build script) due to 1 previous error                                      
[P2] warning: build failed, waiting for other jobs to finish...                                                         
[P2] error: linking with `link.exe` failed: exit code: 1181
[P2]   |                                                                                                                
[P2]   = note: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64\\link.exe" "/NOLOGO" "C:\\Users\\MELVIN~1\\AppData\\Local\\Temp\\rustcIocMnS\\symbols.o" "<2 object files omitted>" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib/{libstd-*,libpanic_unwind-*,libcfg_if-*,libwindows_targets-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libunwind-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "kernel32.lib" "kernel32.lib" "kernel32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:\\git-repos\\github\\minimal-repros/shared-target\\debug\\build\\quote-07adcfe4ff960e36\\build_script_build-07adcfe4ff960e36.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:<sysroot>\\lib\\rustlib\\etc\\libstd.natvis"
[P2]   = note: some arguments are omitted. use `--verbose` to show all linker arguments                                 
[P2]   = note: LINK : fatal error LNK1181: cannot open input file 'D:\git-repos\github\minimal-repros\shared-target\debug\build\quote-07adcfe4ff960e36\build_script_build-07adcfe4ff960e36.build_script_build.d04ca6b7acf63c67-cgu.0.rcgu.o'ΓÉì
[P2]                                                                                                                    
[P2] System.Management.Automation.RemoteException                                                                       
[P2] error: could not compile `quote` (build script) due to 1 previous error                                            
[P2] Exit code: 101                                                                                                     
Builds finished. Exit codes -> P1: 0, P2: 101

Also fails on nightly-2025-12-04:

Set CARGO_TARGET_DIR to D:\git-repos\github\minimal-repros/shared-target
Cleaning CARGO_TARGET_DIR...
Starting builds...
Using cargo toolchain override: nightly-2025-12-04                                                                      
Cargo command: cargo check                                                                                              
[P1] Running: cargo +nightly-2025-12-04 check --manifest-path D:\git-repos\github\minimal-repros\package-a\Cargo.toml   
[P1] info: syncing channel updates for 'nightly-2025-12-04-x86_64-pc-windows-msvc'                                      
[P1] info: latest update on 2025-12-04, rust version 1.93.0-nightly (83e49b75e 2025-12-03)                              
[P1] info: downloading component 'cargo'                                                                                
[P1] info: downloading component 'clippy'                                                                               
[P1] info: downloading component 'rust-docs'                                                                            
[P1] info: downloading component 'rust-std'                                                                             
[P1] info: downloading component 'rustc'                                                                                
[P1] info: downloading component 'rustfmt'                                                                              
[P1] info: installing component 'cargo'                                                                                 
[P1] info: installing component 'clippy'                                                                                
[P1] info: installing component 'rust-docs'                                                                             
[P1] info: installing component 'rust-std'
[P1] info: installing component 'rustc'                                                                                 
[P1] info: installing component 'rustfmt'                                                                               
[P1]    Compiling proc-macro2 v1.0.103                                                                                  
[P1]     Checking unicode-ident v1.0.22                                                                                 
[P1]    Compiling quote v1.0.42                                                                                         
[P1]     Checking syn v2.0.111                                                                                          
[P1]     Checking package-a v0.1.0 (D:\git-repos\github\minimal-repros\package-a)                                       
[P1]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.76s
[P1] Exit code: 0                                                                                                       
[P2] Running: cargo +nightly-2025-12-04 check --manifest-path D:\git-repos\github\minimal-repros\package-b\Cargo.toml   
[P2] info: syncing channel updates for 'nightly-2025-12-04-x86_64-pc-windows-msvc'                                      
[P2] info: latest update on 2025-12-04, rust version 1.93.0-nightly (83e49b75e 2025-12-03)                              
[P2] info: downloading component 'cargo'                                                                                
[P2] error: component download failed for cargo-x86_64-pc-windows-msvc: could not rename downloaded file from 'D:\.tools\.rustup\downloads\4cbebfb5038bfb8219432335f24f63e50af15319637d6c5e9a6e9c7b30580b19.partial' to 'D:\.tools\.rustup\downloads\4cbebfb5038bfb8219432335f24f63e50af15319637d6c5e9a6e9c7b30580b19': The system cannot find the file specified. (os error 2)
[P2] Exit code: 1                                                                                                       
Builds finished. Exit codes -> P1: 0, P2: 1  

When running with cargo build, it works fine:

🪟 Windows 11 Enterprise(10.0.26200) (pwsh) minimal-repros on  rust-nightly-2025-11-21-bug
at 05:00:21 PM ❯ .\repro.ps1 -CargoCommand build -CargoToolchain 1.93.0-beta.1
Set CARGO_TARGET_DIR to D:\git-repos\github\minimal-repros/shared-target
Cleaning CARGO_TARGET_DIR...
Starting builds...
Using cargo toolchain override: 1.93.0-beta.1                                                                           
Cargo command: cargo build                                                                                              
[P1] Running: cargo +1.93.0-beta.1 build --manifest-path D:\git-repos\github\minimal-repros\package-a\Cargo.toml        
[P1]    Compiling proc-macro2 v1.0.103                                                                                  
[P1]    Compiling quote v1.0.42                                                                                         
[P1]    Compiling unicode-ident v1.0.22                                                                                 
[P1]    Compiling syn v2.0.111                                                                                          
[P1]    Compiling package-a v0.1.0 (D:\git-repos\github\minimal-repros\package-a)                                       
[P1]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.17s                                            
[P1] Exit code: 0
[P2] Running: cargo +1.93.0-beta.1 build --manifest-path D:\git-repos\github\minimal-repros\package-b\Cargo.toml        
[P2]     Blocking waiting for file lock on artifact directory                                                           
[P2]    Compiling package-b v0.1.0 (D:\git-repos\github\minimal-repros\package-b)                                       
[P2]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.19s                                            
[P2] Exit code: 0                                                                                                       
Builds finished. Exit codes -> P1: 0, P2: 0

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.93.0-beta.1 (2c283a9a5 2025-12-04)
release: 1.93.0-beta.1
commit-hash: 2c283a9a5c5968eeb9a8f12313f04feb1ff8dfac
commit-date: 2025-12-04
host: x86_64-pc-windows-msvc
libgit2: 1.9.1 (sys:0.20.2 vendored)
libcurl: 8.17.0-DEV (sys:0.4.84+curl-8.17.0 vendored ssl:Schannel)
os: Windows 10.0.26200 (Windows 11 Enterprise) [64-bit]

cargo 1.94.0-nightly (2c283a9a5 2025-12-04)
release: 1.94.0-nightly
commit-hash: 2c283a9a5c5968eeb9a8f12313f04feb1ff8dfac
commit-date: 2025-12-04
host: x86_64-pc-windows-msvc
libgit2: 1.9.1 (sys:0.20.2 vendored)
libcurl: 8.17.0-DEV (sys:0.4.84+curl-8.17.0 vendored ssl:Schannel)
os: Windows 10.0.26200 (Windows 11 Enterprise) [64-bit]

[weihanglo]

Hmm... the fix of #16297 has got into 1.93-beta. Anyway you already reproduced it on nightly.

cc @ranger-ross

[weihanglo]

Sharing the target directory is not a blessed path, but I guess we may want to put the entire #16230 behind -Zbuild-dir-new-layout to not break people sharing the target directory.

@wmmc88 if you also enable the unstable flag -Zbuild-dir-new-layout, then it has no issue I assume?

[ranger-ross]

Hmmm I don't think that adding -Zbuild-dir-new-layout will fix this.
In theory check should still be locking the entire build-dir when -Zbuild-dir-new-layout is not present. So by default 2 cargo check commands should still run in serial on nightly+beta unless the CARGO_BUILD_BUILD_DIR is set to different locations for each build.

[ranger-ross]

Errr I think I see the issue...

We introduced a must_take_artifact_dir_lock that conditionally takes the artifact-dir lock

cargo/src/cargo/core/compiler/layout.rs

Line 130 in e6b0a65

must_take_artifact_dir_lock: bool,

However if artifact-dir == build-dir we only take artifact-dir's lock. But I forgot to update the logic to consider this new flag :(

cargo/src/cargo/core/compiler/layout.rs

Lines 154 to 163 in e6b0a65

	let build_dir_lock = if root == build_root || is_on_nfs_mount(build_root.as_path_unlocked())
	{
	None
	} else {
	Some(build_dest.open_rw_exclusive_create(
	".cargo-lock",
	ws.gctx(),
	"build directory",
	)?)
	};

I'll raise a fix PR. Will probably need your help in the back-porting process

[weihanglo]

Thanks for the quick investigation!

[ranger-ross]

I posted a fix in #16385.

I did not test this on the reproduction repo as I don't have a windows machine handy.
If either of y'all have the ability to build and test on the reproduction repo to double check, I'd appreciate it :)

Though it think its very likely that this is the issue you are facing.

[wmmc88]

@weihanglo I confirmed that -Zbuild-dir-new-layout does not fix this issue

@ranger-ross How do you typically test cargo changes? Will cloning and building cargo itself interfere if I'm using rustup?

[weihanglo]

How do you typically test cargo changes? Will cloning and building cargo itself interfere if I'm using rustup?

Build the pull request and rustup run beta|nightly target/release/cargo <args>

[wmmc88]

I've tested with my repro case and it works!