Reduce cargo's testsuite overhead

[ehuss]

Cargo's testsuite has fairly significant overhead, particularly in disk space usage. For x86_64-pc-windows-gnu it is generating nearly 9.5GB of data. This will eventually run the risk of running out of space on CI and leads to slower tests.

This issue is for tracking ideas and work to try to reduce that overhead. Some ideas:

• Switch tests to default with debuginfo off. Most tests don't care about debuginfo, and it generates a significantly large amount of data on some platforms. Other profile options may also be considered (for example, enable_mac_dsym sets split-debuginfo to "packed". Something like that could be considered for other targets or settings.)

• Switch tests to avoid running cargo build and instead use lighter-weight commands like cargo check or cargo tree. Lots of tests were written before these other commands existed, and the tests themselves aren't particularly interested in the compiled artifacts. This may be difficult and tedious, and may not have significant wins. It also has a moderate risk of inadvertently changing the intent of the test, or otherwise degrading our test coverage. Done in Switch some tests from build to check #11725

• Clean tests after they run. Unfortunately this is quite difficult (or maybe impossible). Particularly on Windows, there can be locking issues (see cargo_test: remove test roots after success #9701, windows: testsuite is unreasonably slow #9585, Revert test directory cleaning change. #7042 and others for more details). There could be workarounds, such as ignoring errors while deleting. Another issue is that for development, it is very useful (to me at least) to keep the directories around. This allows manually entering the directory to test and debug issues. This could also be alleviated with something like an environment variable to toggle the behavior, but that would make running tests more complicated and cumbersome.

• Somehow reduce excess registry overhead. I've noticed that many cargo-add and similar tests are generating very large registries (1.5MB to 2MB each). Ideally those should only be generating the minimal amount of data needed (I would expect about 10x less).

[ChrisDenton]

Clean tests after they run

Idea: A test directory could simply be renamed when it's no longer required. This would allow a cleanup thread or process to somehow be signalled to attempt deletes at its own pace without holding up other work. If there is a setting to keep the test directory then maybe the cleanup thread/process wouldn't be run or would just be a no-op.

Admittedly this adds some complexity.

[weihanglo]

Terribly bad idea: What about a layer of in-memory file system, so we no more suffer from insufficient disk space? It drops after a test finishes. Presumably for registry index and target-dir we could do that. Then we need a larger RAM 😆.

[klensy]

Switch tests to default with debuginfo off. Most tests don't care about debuginfo, and it generates a significantly large amount of data on some platforms. Other profile options may also be considered (for example, enable_mac_dsym sets split-debuginfo to "packed". Something like that could be considered for other targets or settings.)

Stripping debuginfo gives nice size reduction, for example rustc ui tests rust-lang/rust#98140