Fuzz Testing
The fuzzing example demonstrates how to fuzz test Soroban contracts with cargo-fuzz and customize the input to fuzz tests with the arbitrary crate. It also demonstrates how to adapt fuzz tests into reusable property tests with the proptest and proptest-arbitrary-interop crates. It builds on the timelock example.
Run the Example
First go through the setup process to get your development environment configured, then clone the v22.0.1 tag of soroban-examples repository:
git clone -b v22.0.1 https://github.com/stellar/soroban-examples
You will also need the cargo-fuzz tool, and to run cargo-fuzz you will need a nightly Rust toolchain:
cargo install cargo-fuzz
rustup install nightly
To run one of the fuzz tests, navigate to the fuzzing directory and run the cargo fuzz subcommand with the nightly toolchain:
cd fuzzing
cargo +nightly fuzz run fuzz_target_1
If you're developing on MacOS you may need to add the --sanitizer=thread flag in order to fix some known linking errors.
You should see output that begins like this:
$ cargo +nightly fuzz run fuzz_target_1
Compiling soroban-fuzzing-contract v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing)
Compiling soroban-fuzzing-contract-fuzzer v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz)
Finished release [optimized + debuginfo] target(s) in 23.74s
Finished release [optimized + debuginfo] target(s) in 0.07s
Running `fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1 ...`
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 886588732
INFO: Loaded 1 modules (1093478 inline 8-bit counters): 1093478 [0x55eb8e2c7620, 0x55eb8e3d2586),
INFO: Loaded 1 PC tables (1093478 PCs): 1093478 [0x55eb8e3d2588,0x55eb8f481be8),
INFO: 105 files found in /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/corpus/fuzz_target_1
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 105 min: 32b max: 61b total: 3558b rss: 86Mb
#2 pulse ft: 8355 exec/s: 1 rss: 307Mb
#4 pulse cov: 8354 ft: 11014 corp: 1/32b exec/s: 2 rss: 313Mb
#8 pulse cov: 8495 ft: 12420 corp: 4/128b exec/s: 4 rss: 315Mb
The rest of this tutorial will explain how to set up this fuzz test, interpret this output, and remedy fuzzing failures.
Background: Fuzz Testing and Rust
Fuzzing is a kind of testing where new inputs are repeatedly fed into a program in hopes of finding unexpected bugs. This style of testing is commonly employed to increase confidence in the correctness of security-sensitive software.
In Rust, fuzzing is most often performed with the cargo-fuzz tool, which drives LLVM's libfuzzer, though other fuzzing tools are available.
Soroban has built-in support for fuzzing Soroban contracts with cargo-fuzz.
cargo-fuzz is a mutation-based fuzzer: it runs a test program, passing it generated input; while the program is executing, the fuzzer monitors which branches the program takes, and which functions it executes; after execution the fuzzer uses this information to make decisions about how to mutate the previously-used input to create new input that might discover more branches and functions; it then runs the test again with new input, repeating this process for potentially millions of iterations. In this way cargo-fuzz is able to automatically explore execution paths through the program that may never be seen by other types of tests.
If a fuzz tests panics or hard-crashes, cargo-fuzz considers it a failure and provides instructions for repeating the test with the failing inputs.
Fuzz testing is typically an exploratory and interactive process, with the programmer devising schemes for producing input that will stress the program in interesting ways, observing the behavior of the fuzz test, and iterating on the test itself.
Resolving a fuzz testing failure typically involves capturing the problematic input in a unit test. The fuzz test itself may or may not be kept, depending on determinations about the cost of maintaining the fuzzer vs the likelihood of it continuing to find bugs in the future.
While fuzzing non-memory-safe software tends to be more lucrative than fuzzing Rust software, it is still relatively common to find panics and other logic errors in Rust through fuzzing.
In Rust, multiple fuzzers are maintained by the rust-fuzz GitHub organization, which also maintains a "trophy case" of Rust bugs found through fuzzing.
About the Example
The example used for this tutorial is based on the timelock example program, with some changes to demonstrate fuzzing.
The contract, ClaimableBalanceContract, allows one party to deposit an arbitrary quantity of a token to the contract, specifying additionally: the claimants, addresses that may withdraw from the contract; and the time_bound, a specification of when those claimants may withdraw from the account.
The TimeBound type looks like
#[derive(Clone)]
#[contracttype]
pub struct TimeBound {
pub kind: TimeBoundKind,
pub timestamp: u64,
}
#[derive(Clone)]
#[contracttype]
pub enum TimeBoundKind {
Before,
After,
}
ClaimableBalanceContract has two methods, deposit and claim:
pub fn deposit(
env: Env,
from: Address,
token: Address,
amount: i128,
claimants: Vec<Address>,
time_bound: TimeBound,
);
pub fn claim(
env: Env,
claimant: Address,
amount: i128,
);
deposit may only be successfully called once, after which claim may be called multiple times until the balance is completely drained, at which point the contract becomes dormant and may no longer be used.
Fuzz Testing Setup
For these examples, the fuzz tests have been created for you, but normally you would use the cargo fuzz init command to create a fuzzing project as a subdirectory of the contract under test.
To do that you would navigate to the contract directory, in this case, soroban-examples/fuzzing, and execute
cargo fuzz init
A cargo-fuzz project is its own crate, which lives in the fuzz subdirectory of the crate being tested. This crate has its own Cargo.toml and Cargo.lock, and another subdirectory, fuzz_targets, which contains Rust programs, each its own fuzz test.
Our soroban-examples/fuzzing directory looks like
Cargo.toml- this is the contract's manifestCargo.locksrclib.rs- this is the contract code
fuzz- this is the fuzzing crateCargo.toml- this is fuzzing crate's manifestCargo.lockfuzz_targetsfuzz_target_1.rs- this is a single fuzz testfuzz_target_2.rs
There are special considerations to note in the configuration of both the contract's manifest and the fuzzing crate's manifest.
Within the contract's manifest one must specificy the crate type as both "cdylib" and "rlib":
[package]
name = "soroban-fuzzing-contract"
version = "0.0.0"
authors = ["Stellar Development Foundation <[email protected]>"]
license = "Apache-2.0"
edition = "2021"
publish = false
[lib]
crate-type = ["cdylib", "rlib"]
doctest = false
[features]
testutils = []
In most examples, a Soroban contract will only be a "cdylib", a Rust crate that is compiled to a dynamically loadable wasm module. For fuzzing though, the fuzzing crate needs to be able to link to the contract crate as a Rust library, an "rlib".
Note that cargo has a feature/bug that inhibits LTO of cdylibs when a crate is both a "cdylib" and "rlib". This can be worked around by building the contract with either soroban contract build or cargo rustc --crate-type cdylib instead of the typical cargo build.
The contract crate must also provide the "testutils" feature. When "testutils" is activated, the Soroban SDK's contracttype macro emits additional code needed for running fuzz tests.
Within the fuzzing crate's manifest one must turn on the "testutils" features in both the contract crate and the soroban-sdk crate:
[package]
name = "soroban-fuzzing-contract-fuzzer"
version = "0.0.0"
publish = false
edition = "2021"
[package.metadata]
cargo-fuzz = true
[dependencies]
libfuzzer-sys = "0.4"
soroban-sdk = { version = "20.0.0", features = ["testutils"] }
[dependencies.soroban-fuzzing-contract]
path = ".."
features = ["testutils"]
A Simple Fuzz Test
First let's look at fuzz_target_1.rs. This fuzz test does two things: it first deposits an arbitrary amount, then it claims an arbitrary amount.
Again, you can run this fuzzer from the soroban-examples/fuzzing directory with the following command:
cargo +nightly fuzz run fuzz_target_1
The entry point and setup code for Soroban contract fuzz tests will typically look like:
#[derive(Arbitrary, Debug)]
struct Input {
deposit_amount: i128,
claim_amount: i128,
}
fuzz_target!(|input: Input| {
let env = Env::default();
env.mock_all_auths();
env.ledger().set(LedgerInfo {
timestamp: 12345,
protocol_version: 1,
sequence_number: 10,
network_id: Default::default(),
base_reserve: 10,
});
// Turn off the CPU/memory budget for testing.
env.budget().reset_unlimited();
// ... do fuzzing here ...
}
Instead of a main function, cargo-fuzz uses a special entry point defined by the fuzz_target! macro. This macro accepts a Rust closure that accepts input, any Rust type that implements the Arbitrary trait. Here we have defined a struct, Input, that derives Arbitrary.
cargo-fuzz will be responsible for generating input and repeatedly calling this closure.
To test a Soroban contract, we must set up an Env. Note that we have disabled the CPU and memory budget: this will allow us to fuzz arbitrarily complex code paths without worrying about running out of budget; we can assume that running out of budget during a transaction always correctly fails, canceling the transaction; it is not something we need to fuzz.
Refer to the fuzz_target_1.rs source code for additional setup for this contract.
This fuzzer performs two steps: deposit, then claim:
// Deposit, then assert invariants.
{
let _ = timelock_client.try_deposit(
&depositor_address,
&token_contract_id,
&input.deposit_amount,
&vec![
&env,
claimant_address.clone(),
],
&TimeBound {
kind: TimeBoundKind::Before,
timestamp: 123456,
},
);
assert_invariants(
&env,
&timelock_contract_id,
&token_client,
&input
);
}
// Claim, then assert invariants.
{
let _ = timelock_client.try_claim(
&claimant_address,
&input.claim_amount,
);
assert_invariants(
&env,
&timelock_contract_id,
&token_client,
&input
);
}
There are a number of potential strategies for writing fuzz tests. The strategy in this test is to make arbitrary, possibly weird and unrealistic, calls to the contract, disregarding whether those calls succeed or fail, and then to make assertions about the state of the contract.
Because there are many potential failure cases for any given contract call, we don't want to write a fuzz test by attempting to interpret the success or failure of any given call: that path leads to duplicating the contract's logic within the fuzz test. Instead we just want to ensure that, regardless of what happened during execution, the contract is never left in an invalid state.
Notice the use of the try_ client function to invoke the contract. Each contract function can be invoked with a try_ variant, which captures any errors, including panics and crashes, and returns the value on success or an error otherwise. Without using the try_ variant, a panic from within a contract will immediately cause the fuzz test to fail, but in most cases a panic within a contract does not indicate a bug - it is simply how a Soroban contract cancels a transaction. try_ returns a Result, but here we discard it.
Finally, the assert_invariants function is where we make any assertions we can about the state of the contract:
/// Directly inspect the contract state and make assertions about it.
fn assert_invariants(
env: &Env,
timelock_contract_id: &Address,
token_client: &TokenClient,
input: &Input,
) {
// Configure the environment to access the timelock contract's storage.
env.as_contract(timelock_contract_id, || {
let storage = env.storage();
// Get the two datums owned by the timelock contract.
let is_initialized = storage.has(&DataKey::Init);
let claimable_balance = storage.get::<_, ClaimableBalance>(&DataKey::Balance);
// Call the token client to get the balance held in the timelock contract.
// This consumes contract execution budget.
let actual_token_balance = token_client.balance(timelock_contract_id);
// There can only be a claimaible balance after the contract is initialized,
// but once the balance is claimed there is no balance,
// but the contract remains initialized.
// This is a truth table of valid states.
assert!(match (is_initialized, claimable_balance.is_some()) {
(false, false) => true,
(false, true) => false,
(true, true) => true,
(true, false) => true,
});
assert!(actual_token_balance >= 0);
if let Some(claimable_balance) = claimable_balance {
let claimable_balance = claimable_balance.expect("balance");
assert!(claimable_balance.amount > 0);
assert!(claimable_balance.amount <= input.deposit_amount);
assert_eq!(claimable_balance.amount, actual_token_balance);
assert!(claimable_balance.claimants.len() > 0);
}
});
}
Interpreting cargo-fuzz Output
If you run cargo-fuzz with fuzz_target_1, from inside the soroban-examples/fuzzing directory, you will see output similar to:
$ cargo +nightly fuzz run fuzz_target_1
Compiling soroban-fuzzing-contract v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing)
Compiling soroban-fuzzing-contract-fuzzer v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz)
Finished release [optimized + debuginfo] target(s) in 25.18s
Finished release [optimized + debuginfo] target(s) in 0.08s
Running `fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1 -artifact_prefix=/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/artifacts/fuzz_target_1/ /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/corpus/fuzz_target_1`
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1384064486
INFO: Loaded 1 modules (1122058 inline 8-bit counters): 1122058 [0x561f6ecd4fc0, 0x561f6ede6eca),
INFO: Loaded 1 PC tables (1122058 PCs): 1122058 [0x561f6ede6ed0,0x561f6ff05f70),
INFO: 173 files found in /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/corpus/fuzz_target_1
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 173 min: 32b max: 61b total: 6039b rss: 83Mb
#4 pulse cov: 4848 ft: 10214 corp: 1/32b exec/s: 2 rss: 313Mb
#8 pulse cov: 8507 ft: 11743 corp: 4/128b exec/s: 4 rss: 315Mb
#16 pulse cov: 8512 ft: 12393 corp: 10/320b exec/s: 8 rss: 319Mb
thread '<unnamed>' panicked at 'assertion failed: claimable_balance.amount > 0', fuzz_targets/fuzz_target_1.rs:130:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
==6102== ERROR: libFuzzer: deadly signal
#0 0x561f6ae3a431 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1c80431) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#1 0x561f6e3855b0 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x51cb5b0) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#2 0x561f6e35c08a (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x51a208a) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#3 0x7fce05f5e08f (/lib/x86_64-linux-gnu/libc.so.6+0x4308f) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
#4 0x7fce05f5e00a (/lib/x86_64-linux-gnu/libc.so.6+0x4300a) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
#5 0x7fce05f3d858 (/lib/x86_64-linux-gnu/libc.so.6+0x22858) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
...
#27 0x561f6e3847b9 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x51ca7b9) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#28 0x561f6ad98346 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1bde346) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#29 0x7fce05f3f082 (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
#30 0x561f6ad9837d (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1bde37d) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x5d,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0xff,0x5f,0x5f,0x52,0xff,
\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000]\000\000\000\000\000\000\000\000\377__R\377
artifact_prefix='/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/artifacts/fuzz_target_1/'; Test unit written to /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Base64: AAAAAAAAAAAAAAAAAAAAAAAAXQAAAAAAAAAA/19fUv8=
────────────────────────────────────────────────────────────────────────────────
Failing input:
fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Output of `std::fmt::Debug`:
Input {
deposit_amount: 0,
claim_amount: -901525218878596739118967460911579136,
}
Reproduce with:
cargo fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Minimize test case with:
cargo fuzz tmin fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
────────────────────────────────────────────────────────────────────────────────
Error: Fuzz target exited with exit status: 77
This is a fuzzing failure, indicating a bug in either the fuzzer or the program. The details will be different.
Here is the same output, with less important lines trimmed:
thread '<unnamed>' panicked at 'assertion failed: claimable_balance.amount > 0', fuzz_targets/fuzz_target_1.rs:130:13
...
Failing input:
fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Output of `std::fmt::Debug`:
Input {
deposit_amount: 0,
claim_amount: -901525218878596739118967460911579136,
}
Reproduce with:
cargo fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Minimize test case with:
cargo fuzz tmin fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
The first line here is printed by our Rust program, and indicates exactly where the fuzzer panicked. The later lines indicate how to reproduce this failing case.
The first thing to do when you get a fuzzing failure is copy the command to reproduce the failure, so that you can use it to debug:
cargo +nightly fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-04704b1542f61a21a4649e39023ec57ff502f627
Notice though that we need to tell cargo to use the nightly toolchain with the +nightly flag, something that cargo-fuzz doesn't print in its version of the command.
Another thing to notice is that by default, cargo-fuzz / libfuzzer does not print names of functions in its output, as in the stack trace:
==6102== ERROR: libFuzzer: deadly signal
#0 0x561f6ae3a431 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1c80431) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
...
#28 0x561f6ad98346 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1bde346) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
#29 0x7fce05f3f082 (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
#30 0x561f6ad9837d (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1bde37d) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
Depending on how your system is set up, you may or may not have this problem. In order to print stack traces, libfuzzer needs the llvm-symbolizer program. On Ubuntu-based systems this can be installed with the llvm-dev package:
sudo apt install llvm-dev
After which libfuzzer will print demangled function names instead of addresses:
==6323== ERROR: libFuzzer: deadly signal
#0 0x557c9da6a431 in __sanitizer_print_stack_trace /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
#1 0x557ca0fb55b0 in fuzzer::PrintStackTrace() /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.5/libfuzzer/FuzzerUtil.cpp:210:38
#2 0x557ca0f8c08a in fuzzer::Fuzzer::CrashCallback() /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.5/libfuzzer/FuzzerLoop.cpp:233:18
#3 0x557ca0f8c08a in fuzzer::Fuzzer::CrashCallback() /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.5/libfuzzer/FuzzerLoop.cpp:228:6
#4 0x7ff19e84d08f (/lib/x86_64-linux-gnu/libc.so.6+0x4308f) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
#5 0x7ff19e84d00a in __libc_signal_restore_set /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/internal-signals.h:86:3
#6 0x7ff19e84d00a in raise /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:48:3
#7 0x7ff19e82c858 in abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79:7
...
#23 0x557c9daee89a in fuzz_target_1::assert_invariants::hd6d4f9549b01c31c /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/fuzz_targets/fuzz_target_1.rs:103:5
#24 0x557c9daee89a in fuzz_target_1::_::run::hac1117cb3dfecb2b /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/fuzz_targets/fuzz_target_1.rs:69:9
#25 0x557c9daecea6 in rust_fuzzer_test_input /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.5/src/lib.rs:297:60
...
#37 0x557c9d9c8346 in main /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libfuzzer-sys-0.4.5/libfuzzer/FuzzerMain.cpp:20:30
#38 0x7ff19e82e082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#39 0x557c9d9c837d in _start (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1+0x1bde37d) (BuildId: 6a95a932984a405ebab8171dddc9f812fdf16846)
To continue, our program has a bug that should be easy to fix by inspecting the error and making a slight modification to the source.
Once the bug is fixed, the fuzzer will run continuously, producing output that looks like
$ cargo +nightly fuzz run fuzz_target_1
Compiling soroban-fuzzing-contract v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing)
Compiling soroban-fuzzing-contract-fuzzer v0.0.0 (/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz)
Finished release [optimized + debuginfo] target(s) in 24.91s
Finished release [optimized + debuginfo] target(s) in 0.08s
Running `fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1 -artifact_prefix=/home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/artifacts/fuzz_target_1/ /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/corpus/fuzz_target_1`
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1619748028
INFO: Loaded 1 modules (1122061 inline 8-bit counters): 1122061 [0x5647a55b9080, 0x5647a56caf8d),
INFO: Loaded 1 PC tables (1122061 PCs): 1122061 [0x5647a56caf90,0x5647a67ea060),
INFO: 173 files found in /home/azureuser/data/stellar/soroban-examples/fuzzing/fuzz/corpus/fuzz_target_1
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 173 min: 32b max: 61b total: 6039b rss: 85Mb
#2 pulse ft: 8067 exec/s: 1 rss: 312Mb
#4 pulse cov: 8068 ft: 10709 corp: 1/32b exec/s: 2 rss: 315Mb
#8 pulse cov: 8476 ft: 11498 corp: 5/160b exec/s: 4 rss: 317Mb
#16 pulse cov: 8512 ft: 12362 corp: 9/288b exec/s: 8 rss: 320Mb
#32 pulse cov: 8516 ft: 13290 corp: 19/608b exec/s: 10 rss: 326Mb
#64 pulse cov: 8516 ft: 13311 corp: 27/864b exec/s: 21 rss: 340Mb
#128 pulse cov: 8540 ft: 13536 corp: 37/1196b exec/s: 25 rss: 365Mb
#175 INITED cov: 8540 ft: 13580 corp: 42/1387b exec/s: 29 rss: 382Mb
#177 NEW cov: 8545 ft: 13821 corp: 43/1419b lim: 48 exec/s: 29 rss: 384Mb L: 32/48 MS: 1 ChangeASCIIInt-
#178 NEW cov: 8545 ft: 13824 corp: 44/1451b lim: 48 exec/s: 29 rss: 384Mb L: 32/48 MS: 1 ChangeBinInt-
#229 NEW cov: 8545 ft: 13826 corp: 45/1483b lim: 48 exec/s: 38 rss: 401Mb L: 32/48 MS: 1 ChangeByte-
#256 pulse cov: 8545 ft: 13826 corp: 45/1483b lim: 48 exec/s: 36 rss: 410Mb
#361 NEW cov: 8545 ft: 13830 corp: 46/1521b lim: 48 exec/s: 40 rss: 451Mb L: 38/48 MS: 5 ShuffleBytes-CMP-EraseBytes-CopyPart-ChangeBinInt- DE: "\005\000\000\000"-
NEW_FUNC[1/1]: 0x5647a2964640 in rand::rngs::adapter::reseeding::ReseedingCore$LT$R$C$Rsdr$GT$::reseed_and_generate::ha760ded93293681c /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.7.3/src/rngs/adapter/reseeding.rs:235
#368 NEW cov: 8557 ft: 13842 corp: 47/1566b lim: 48 exec/s: 40 rss: 454Mb L: 45/48 MS: 2 CrossOver-InsertRepeatedBytes-
#512 pulse cov: 8557 ft: 13842 corp: 47/1566b lim: 48 exec/s: 46 rss: 502Mb
#850 NEW cov: 8557 ft: 13843 corp: 48/1610b lim: 48 exec/s: 53 rss: 591Mb L: 44/48 MS: 2 CopyPart-ChangeBit-
#1024 pulse cov: 8557 ft: 13843 corp: 48/1610b lim: 48 exec/s: 56 rss: 645Mb
#1796 NEW cov: 8557 ft: 13863 corp: 49/1642b lim: 53 exec/s: 71 rss: 669Mb L: 32/48 MS: 1 ChangeBinInt-
#1913 NEW cov: 8557 ft: 13864 corp: 50/1675b lim: 53 exec/s: 73 rss: 669Mb L: 33/48 MS: 2 ShuffleBytes-InsertByte-
#3749 REDUCE cov: 8557 ft: 13864 corp: 50/1670b lim: 68 exec/s: 98 rss: 669Mb L: 39/48 MS: 1 EraseBytes-
...
And this output will continue until the fuzzer is killed with Ctrl-C.
Next, let's look at a single line of fuzzer output:
#177 NEW cov: 8545 ft: 13821 corp: 43/1419b lim: 48 exec/s: 29 rss: 384Mb L: 32/48 MS: 1 ChangeASCIIInt-
The most important column here is cov. This is a cumulative measure of branches covered by the fuzzer. When this number stops increasing the fuzzer has probably explored as much of the program as it can. The other columns are described in the libfuzzer documentation.
Finally, lets look at this warning:
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes.
By default, libfuzzer only generates input up to 4096 bytes. In a lot of cases, this is probably reasonable, but cargo-fuzz can increase the max_len by appending the argument after --:
cargo +nightly fuzz run fuzz_target_1 -- -max_len=20000
All the options to libfuzzer can be listed with
cargo +nightly fuzz run fuzz_target_1 -- -help=1
See the libfuzzer documentation for more.
Accepting Soroban Types as Input with the SorobanArbitrary Trait
Inputs to the fuzz_target! macro must implement the Arbitrary trait, which accepts bytes from the fuzzer driver and converts them to Rust values. Soroban types though are managed by the host environment, and so must be created from an Env value, which is not available to the fuzzer driver. The SorobanArbitrary trait, implemented for all Soroban contract types, exists to bridge this gap: it defines a prototype pattern whereby the fuzz_target macro creates prototype values that the fuzz program can convert to contract values with the standard soroban conversion traits, FromVal or IntoVal.
The types of prototypes are identified by the associated type, SorobanArbitrary::Prototype:
pub trait SorobanArbitrary:
TryFromVal<Env, Self::Prototype> + IntoVal<Env, Val> + TryFromVal<Env, Val>
{
type Prototype: for <'a> Arbitrary<'a>;
}
Types that implement SorobanArbitrary include:
i32,u32,i64,u64,i128,u128,I256,U256,(), andbool,Error,Bytes,BytesN,Vec,Map,Address,Symbol,Val,
All user-defined contract types, those with the contracttype attribute, automatically derive SorobanArbitrary. Note that SorobanArbitrary is only derived when the "testutils" Cargo feature is active. This implies that, in general, to make a Soroban contract fuzzable, the contract crate must define a "testutils" Cargo feature, that feature should turn on the "soroban-sdk/testutils" feature, and the fuzz test, which is its own crate, must turn that feature on.
A More Complex Fuzz Test
The fuzz_target_2.rs example, demonstrates the use of SorobanArbitrary, the advancement of time, and more advanced fuzzing techniques.
This fuzz test takes a much more complex input, where some of the values are user-defined types exported from the contract under test. This test is structured as a simple interpreter, where the fuzzing harness provides arbitrarily-generated "steps", where each step is either a deposit command or a claim command. The test then treats each of these steps as a separate transaction: it maintains a snapshot of the blockchain state, and for each step creates a fresh environment in which to execute the contract call, simulating the advancement of time between each step. As in the previous example, assertions are made after each step.
The input to the fuzzer looks, in part, like:
#[derive(Arbitrary, Debug)]
struct Input {
addresses: [<Address as SorobanArbitrary>::Prototype; NUM_ADDRESSES],
#[arbitrary(with = |u: &mut Unstructured| u.int_in_range(0..=i128::MAX))]
token_mint: i128,
steps: RustVec<Step>,
}
#[derive(Arbitrary, Debug)]
struct Step {
#[arbitrary(with = |u: &mut Unstructured| u.int_in_range(1..=u64::MAX))]
advance_time: u64,
command: Command, // `Command` not shown here - see the full source.
}
This shows how to use the SorobanArbitrary::Prototype associated type to define inputs to the fuzzer. A Soroban Address can only be created with an Env, so cannot be generated directly by the Arbitrary trait. Instead we use the fully-qualified name of the Address prototype, <Address as SorobanArbitrary>::Prototype, to ask for Address's prototype instead. Then when our fuzzer needs the Address we instantiate it with the FromVal trait:
let depositor_address = Address::from_val(&env, &input.addresses[cmd.depositor_index]);
The contract we are fuzzing is a timelock contract, where calculation of time is crucial for correctness. So our testing must account for the advancement of time.
The contract defines a TimeBound type and accepts it in the deposit method:
#[derive(Clone, Debug)]
#[contracttype]
pub struct TimeBound {
pub kind: TimeBoundKind,
pub timestamp: u64,
}
#[contractimpl]
impl ClaimableBalanceContract {
pub fn deposit(
env: Env,
from: Address,
token: Address,
amount: i128,
claimants: Vec<Address>,
time_bound: TimeBound,
) {
...
}
}
In our fuzzer, one of the possible commands issued each step is a DepositCommand:
#[derive(Arbitrary, Debug)]
struct DepositCommand {
#[arbitrary(with = |u: &mut Unstructured| u.int_in_range(0..=NUM_ADDRESSES - 1))]
depositor_index: usize,
amount: i128,
// This is an ugly way to get a vector of integers in range
#[arbitrary(with = |u: &mut Unstructured| {
u.arbitrary_len::<usize>().map(|len| {
(0..len).map(|_| {
u.int_in_range(0..=NUM_ADDRESSES - 1)
}).collect::<Result<RustVec<usize>, _>>()
}).and_then(|inner_result| inner_result)
})]
claimant_indexes: RustVec<usize>,
time_bound: <TimeBound as SorobanArbitrary>::Prototype,
}
Notice that this command again uses the SorobanArbitrary::Prototype associated type to accept a TimeBound as input.
To advance time we maintain a LedgerSnapshot, defined in the soroban-ledger-snapshot crate. For each step we call Env::from_snapshot to create a fresh environment to execute the step, then Env::to_snapshot to create a new snapshot to use in the following step.
Here is a simplified outline of how this works. See the full source code for details.
let init_snapshot = {
let init_ledger = LedgerInfo {
timestamp: 12345,
protocol_version: 1,
sequence_number: 10,
network_id: Default::default(),
base_reserve: 10,
min_temp_entry_ttl: u32::MAX,
min_persistent_entry_ttl: u32::MAX,
};
LedgerSnapshot::from(init_ledger, None)
};
let mut prev_env = Env::from_snapshot(init_snapshot);
for step in &config.input.steps {
// Advance time and create a new env from snapshot.
let curr_env = {
let mut snapshot = prev_env.to_snapshot();
snapshot.sequence_number += 1;
snapshot.timestamp = snapshot.timestamp.saturating_add(step.advance_time);
let env = Env::from_snapshot(snapshot);
env.budget().reset_unlimited();
env
};
step.command.exec(&config, &curr_env);
prev_env = curr_env;
}
Converting a Fuzz Test to a Property Test
In addition to fuzz testing, Soroban supports property testing in the style of quickcheck, by using the proptest and proptest-arbitrary-interop crates in conjunction with the SorobanArbitrary trait.
Property tests are similar to fuzz tests in that they generate randomized input. Property tests though do not instrument their test cases or mutate their input based on feedback from previous tests. Thus they are a weaker form of test.
The great benefit of property tests though is that they can be included in standard Rust test suites and require no extra tooling to execute. One might take advantage of this by interactively fuzzing to discover deep bugs, then convert fuzz tests to property tests to help prevent regressions.
The proptest.rs file is a translation of fuzz_target_1.rs to a property test.