Mutation Testing

Mutation testing is making changes to a program, either manually or automatically, to identify changes that can be made that don't get caught by tests.

Mutation testing is similar to measuring code coverage, sharing the same goal to identify code not covered by tests. But where code coverage focuses on checking if a line of code is executed during a test, mutation testing will actually check that a test fails when the line is changed. A line of code can look like it is covered by tests, but the outcomes and side-effects may not be asserted on in the tests.

How to do Mutation Testing

The cargo-mutants tool can be used to automatically and iteratively modify the Rust code, and rerun the tests after each mutation, to identify code not tested.

1. Install cargo-mutants:

   cargo install --locked cargo-mutants

2. Run the cargo mutants command inside your contract's crate directory.

   $ cargo mutants
   Found 4 mutants to test
   ok       Unmutated baseline in 19.0s build + 0.6s test
    INFO Auto-set test timeout to 20s
   MISSED   src/lib.rs:14:9: replace IncrementContract::increment -> u32 with 1 in 0.4s build + 0.4s test
   4 mutants tested in 23s: 1 missed, 3 caught

   Code that is identified as not covered by a test will be outputted as a MISSED line in the output.

   Diffs of each change that was attempted can be found in the mutants.out/diff directory.