Detecting Unexpected Changes with Test Snapshots

Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes.

However tests are limited, as they only show changes to values that the tests assert on.

Test Snapshots

The Soroban Rust SDK generates test snapshots on every test involving an Env. Test snapshots are enabled by default. At the end of the test the Env writes a JSON file to the test_snapshots directory with a full snapshot of all the events published, logged, and the final ledger storage state.

Most tests have a single Env and will result in a single test snapshot. Tests that have multiple Envs will write multiple test snapshots, one for each Env. Test snapshot files are named with a incrementing number on the end to separate the test snapshots for each Env.

How to Use Test Snapshots

1. Write tests using the default Env. For example:

   #![cfg(test)]
   use soroban_sdk::Env;
   
   use crate::{Contract, ContractClient};
   
   #[test]
   fn test_abc() {
       let env = Env::default();
       let contract_id = env.register_contract(None, Contract);
       let client = ContractClient::new(&env, &contract_id);
   
       assert_eq!(client.increment(), 1);
   
       // At the end of the test the Env will automatically write a test snapshot
       // to the following directory: test_snapshots/test_abc.1.json
   }

2. Run the tests and see that the test snapshots have been written to test_snapshots/.

3. Commit the test snapshots to source control.

4. On future updates look out for changes to test snapshots in tests that are unexpected. For example, when changing one part of a contract if the test snapshots for other parts of the contract or unrelated end-to-end tests change, that could signal that side-effects have occurred.

5. As needed, diff test snapshots as needed to look for hints to why the unexpected change may have occurred.

Note that test snapshots are extremely verbose and thoroughly understanding each in isolation may not be realistic. Test snapshots are most useful when changes appear and can be diffed, such as a new event being published, or storage changing.

To give this a go, check out the Getting Started contract or any of the examples, run the tests, and look for the test snapshots on disk.