-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Description
Currently our CLI/UI tests are doing manual assertions with the output, manually parsing JSON and comparing against another manually parsed fixture file etc.
This is not ideal, and not very scalable as any change in the output will also require manually updating all the fixture files.
We should migrate to using the snapbox
library which is also used by cargo
in most of their tests.
Supersedes #7604
Unblocks #6569
The migration can be done gradually over multiple commits, however we also need an initial integration with the TestCommand to allow asserting with
snapbox
'sstr!
andfile!
macros.
Completed in #8406
Migration guide
#8406 added helper assert()
method to TestCommand
, which executes configured command and returns snapbox's OutputAssert
allowing to perform checks on output.
-
For tests which perform assertions on stdout, instead of asserting value of
stdout_lossy
/unchecked_stdout
prefer usingassert().stdout_eq(str![["<fixture>"]])
orassert_success().stdout_eq(str![["<fixture>"]])
if you need to assert successful status code.i.e the following test
let output = cmd.args([...]).stdout_lossy(); assert_eq!(stdout, "0x123");
would become
cmd.args([...]).assert_success().stdout_eq(str![["0x123"]]);
-
For tests using fixture files, prefer inlining those fixtures into
str!
macro invocations and removing the fixture files.i.e such assertion
cmd.unchecked_output().stdout_matches_path( PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("path/to/fixture"), );
would become
cmd.args([...]).assert().stdout_eq(str![["<fixture file contents>"]])
Large fixture files can be kept as is, prefer using
file!
macro for those. -
Prefer replacing all irrelevant info in fixtures with snapbox filters (
...
as line wildcard,[..]
as character-wildcard).
i.e. if we're asserting on traces and output looks like this:Compiling 100 files with Solc 0.8.23 Solc 0.8.23 finished in 14.35s Compiler run successful! Ran 1 test for test/Test.t.sol [PASS] test() (gas: 10000) Traces: [10000] TEST_CONTRACT::test() ├─ [0] VM::assertEq(1, 1) [staticcall] │ └─ ← [Return] └─ ← [Stop]
the test would look similar to:
cmd.arg("test").assert_success().stdout_eq(str![[r#" ... Traces: [10000] TEST_CONTRACT::test() ├─ [0] VM::assertEq(1, 1) [staticcall] │ └─ ← [Return] └─ ← [Stop] "#]]);
-
Note: when writing tests it might be handy to set fixture to empty value (
str![[""]]
) and populate by runningcargo test
withSNAPSHOTS=overwrite
to let snapbox populate fixtures automatically