Differential Tests

Differential testing is the testing of two things to discover differences in their behavior.

The goal is to prove that the two things behave consistently, and that they do not diverge in behavior except for some expected differences. The assertions should be as broad as possible, broadly testing that all observable outcomes do not change, except for any expected changes.

This strategy is effective when building something new that should behave like something that already exists. That could be a new version of a contract that has unchanged behavior from it's previous version. Or it could be the same contract with an updated SDK or other dependency. Or it could be a refactor that expects no functional changes.

This strategy can be used in the context of unit and integration tests, or in the context of fuzz tests as well.

tip

All contracts built with the Rust Soroban SDK have a form of differential testing built-in and enabled by default. See Differential Testing with Test Snapshots.

How to Write Differential Tests

To experiment with writing a differential test, open a contract that you've deployed, or checkout an example from the [soroban-examples] repository and deploy it.

Assuming the contract has been deployed, and changes are being made to the local copy. We need to check that unchanged behavior in the contract hasn't changed compared to what is deployed.

1. Use the stellar contract fetch command to fetch the contract that's already deployed. The contract already deployed will be used as a baseline that the local copy is expected to behave like.

   stellar contract fetch --id C... --out-file contract.wasm

2. Import the contract into the tests with the contractimport! macro.

   mod deployed {
       soroban_sdk::contractimport!(file = "contract.wasm");
   }

3. Write a test that runs the same logic for the deployed contract and the local contract, comparing the result. Assuming the increment example is in use, the test would look something like the following.

   #![cfg(test)]
   use crate::{IncrementContract, IncrementContractClient};
   use soroban_sdk::{testutils::Events as _, Env};
   
   mod deployed {
       soroban_sdk::contractimport!(file = "contract.wasm");
   }
   
   #[test]
   fn differential_test() {
       let env = Env::default();
       assert_eq!(
           // Baseline – the deployed contract
           {
               let contract_id = env.register(deployed::WASM, ());
               let client = IncrementContractClient::new(&env, &contract_id);
               (
                   // Return Values
                   (
                       client.increment(),
                       client.increment(),
                       client.increment(),
                   ),
                   // Events
                   env.events.all(),
               )
           },
           // Local – the changed or refactored contract
           {
               let contract_id = env.register(IncrementContract, ());
               let client = IncrementContractClient::new(&env, &contract_id);
               (
                   // Return Values
                   (
                       client.increment(),
                       client.increment(),
                       client.increment(),
                   ),
                   // Events
                   env.events.all(),
               )
           },
       );
   }

4. Run the test to compare the baseline and local observable outcomes.

This test uses the same patterns used in unit tests and integration tests:

1. Create an environment, the Env.
2. Import the Wasm contract to compare with.
3. Register the local contract to be tested.
4. Invoke functions using a client.
5. Assert equality.

tip

Differential tests work best when less assumptions are made. Rather than asserting only on specific return values or on implementation details like specific state, asserting on all observable outcomes and including things like events published or return values of other read-only contract functions will help to discover unexpected changes.

info

Depending on the test complexity it can be desirable to use an independent Env for testing the deployed vs local. However at the moment it is only possible to compare host values, like String, Bytes, Vec, Map, if they've been created using the same Env. The tracking issue for supportin comparisons across environments is stellar/rs-soroban-sdk#1360.