Skip to content

meta(testing): migrate CLI / UI tests to snapbox #8389

@DaniPopes

Description

@DaniPopes

Currently our CLI/UI tests are doing manual assertions with the output, manually parsing JSON and comparing against another manually parsed fixture file etc.

This is not ideal, and not very scalable as any change in the output will also require manually updating all the fixture files.

We should migrate to using the snapbox library which is also used by cargo in most of their tests.

Supersedes #7604

Unblocks #6569

The migration can be done gradually over multiple commits, however we also need an initial integration with the TestCommand to allow asserting with snapbox's str! and file! macros.

Completed in #8406

Migration guide

#8406 added helper assert() method to TestCommand, which executes configured command and returns snapbox's OutputAssert allowing to perform checks on output.

  • For tests which perform assertions on stdout, instead of asserting value of stdout_lossy/unchecked_stdout prefer using assert().stdout_eq(str![["<fixture>"]]) or assert_success().stdout_eq(str![["<fixture>"]]) if you need to assert successful status code.

    i.e the following test

    let output = cmd.args([...]).stdout_lossy();
    assert_eq!(stdout, "0x123");

    would become

    cmd.args([...]).assert_success().stdout_eq(str![["0x123"]]);
  • For tests using fixture files, prefer inlining those fixtures into str! macro invocations and removing the fixture files.

    i.e such assertion

    cmd.unchecked_output().stdout_matches_path(
        PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("path/to/fixture"),
    );

    would become

    cmd.args([...]).assert().stdout_eq(str![["<fixture file contents>"]])

    Large fixture files can be kept as is, prefer using file! macro for those.

  • Prefer replacing all irrelevant info in fixtures with snapbox filters (... as line wildcard, [..] as character-wildcard).
    i.e. if we're asserting on traces and output looks like this:

    Compiling 100 files with Solc 0.8.23
    Solc 0.8.23 finished in 14.35s
    Compiler run successful!
    
    Ran 1 test for test/Test.t.sol
    [PASS] test() (gas: 10000)
    Traces:
      [10000] TEST_CONTRACT::test()
        ├─ [0] VM::assertEq(1, 1) [staticcall]
        │  └─ ← [Return] 
        └─ ← [Stop] 
    

    the test would look similar to:

    cmd.arg("test").assert_success().stdout_eq(str![[r#"
    ...
    Traces:
      [10000] TEST_CONTRACT::test()
        ├─ [0] VM::assertEq(1, 1) [staticcall]
        │  └─ ← [Return] 
        └─ ← [Stop]
    "#]]);
  • Note: when writing tests it might be handy to set fixture to empty value (str![[""]]) and populate by running cargo test with SNAPSHOTS=overwrite to let snapbox populate fixtures automatically

Metadata

Metadata

Assignees

Labels

A-internalsArea: internalsT-metaType: metafirst issueA good way to start contributing

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions