Differential Tests with Test Snapshots
Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes.
However tests are limited, as they only show changes to values that the tests assert on.
Test snapshots are one tool for performing differential testing. See Differential Testing for other ways.
Test Snapshotsโ
The Soroban Rust SDK generates test snapshots on every test involving an Env
. Test snapshots are enabled by default. At the end of the test the Env
writes a JSON file to the test_snapshots
directory with a full snapshot of all the events published, and the final ledger storage state.
Most tests have a single Env
and will result in a single test snapshot. Tests that have multiple Env
s will write multiple test snapshots, one for each Env
. Test snapshot files are named with a incrementing number on the end to separate the test snapshots for each Env
.
How to Use Test Snapshotsโ
-
Write tests using the default
Env
. For example:#![cfg(test)]
use soroban_sdk::Env;
use crate::{Contract, ContractClient};
#[test]
fn test_abc() {
let env = Env::default();
let contract_id = env.register_contract(None, Contract);
let client = ContractClient::new(&env, &contract_id);
assert_eq!(client.increment(), 1);
// At the end of the test the Env will automatically write a test snapshot
// to the following directory: test_snapshots/test_abc.1.json
} -
Run the tests and see that the test snapshots have been written to
test_snapshots/
. -
Commit the test snapshots to source control.
-
On future updates look out for changes to test snapshots in tests that are unexpected. For example, when changing one part of a contract if the test snapshots for other parts of the contract or unrelated end-to-end tests change, that could signal that side-effects have occurred.
-
Diff test snapshots as needed to look for hints to why an unexpected change has occurred.
Test snapshots files are verbose. Test snapshots are most useful when changes appear and can be diffed, such as a new event being published, or storage changing.
To give this a go, check out the Getting Started contract or any of the examples, run the tests, and look for the test snapshots on disk.
Guides in this category:
๐๏ธ Unit Tests
Unit tests are small tests that test smart contracts.
๐๏ธ Mocking
Mocking dependency contracts in tests.
๐๏ธ Integration Tests
Integration testing uses dependency contracts instead of mocks.
๐๏ธ Fuzzing
Fuzzing and property testing to find unexpected behavior.
๐๏ธ Differential Tests
Differential testing detects unintended changes.
๐๏ธ Differential Tests with Test Snapshots
Differential testing using automatic test snapshots.
๐๏ธ Mutation Testing
Mutation testing finds code not tested.
๐๏ธ Test Authorization
Write tests that test contract authorization.
๐๏ธ Test Events
Write tests that test contract events.