vertigo

Vertigo is a mutation testing framework designed to work specifically for smart contracts. This mutation testing framework implements a range of mutation operators that are either selected from previous works or tailored to solidity.

Quick Start Guide

To install vertigo, execute the following command:

pip3 install --user eth-vertigo

You can now run vertigo on a truffle project with the following command (assuming you have a development network configured in yourtruffle-config.js):

vertigo run --network development

Depending on your environment it might be required to specify the location of the truffle executable:

vertigo run --network development --truffle-location <node_dir>/bin/truffle 

Or, if you're using Hardhat, just use dynamic networks:

vertigo run --hardhat-parallel 8

There are a few additional parameters available that allow you to tweak the execution of vertigo:

$ vertigo run --help                                                                                                                                                                  
Usage: vertigo run [OPTIONS]

  Performs a core test campaign

Options:
  --output TEXT                   Output core test results to file
  --network TEXT                  Network names that vertigo can use
  --ganache-path TEXT             Path to ganache binary
  --ganache-network <TEXT INTEGER>...
                                  Dynamic networks that vertigo can use eg.
                                  (develop, 8485)

  --ganache-network-options TEXT  Options to pass to dynamic ganache networks
  --hardhat-parallel INTEGER      Amount of networks that hardhat should be
                                  using in parallel

  --rules TEXT                    Universal Mutator style rules to use in
                                  mutation testing

  --truffle-location TEXT         Location of truffle cli
  --sample-ratio FLOAT            If this option is set. Vertigo will apply
                                  the sample filter with the given ratio

  --exclude TEXT                  Vertigo won't mutate files in these
                                  directories

  --incremental TEXT              File where incremental mutation state is
                                  stored

  --help                          Show this message and exit.
                                                                                                                                     

Known Issues

Ganache is generally used only for a single run of the entire test suite. For the general use case, it does not matter if Ganache creates a few thousand files. Unfortunately, once you start executing the entire test suite hundreds of times, you can end up with millions of files, and your machine could run out of free inode's. You can check whether this happens to you by running:

df -i

This issue (#1) is known, and we're working on a fix.

In the meanwhile. If your test suite is large enough to munch all your inodes, then there are two options:

• You can use the command line option --sample-ratio to select a random subsample of the mutations (reducing the number of times that the test suite is run)
• You can create a partition that has a sufficient amount of inodes available

Publications and Articles

Practical Mutation Testing for Smart Contracts - Joran J. Honig, Maarten H. Everts, Marieke Huisman

Introduction into Mutation Testing - Joran Honig

Mutation Testing for Smart Contracts - A step by step guide - Joran Honig

If you want to cite vertigo, please use the following:

@InProceedings{10.1007/978-3-030-31500-9_19,
author="Honig, Joran J.
and Everts, Maarten H.
and Huisman, Marieke",
title="Practical Mutation Testing for Smart Contracts",
booktitle="Data Privacy Management, Cryptocurrencies and Blockchain Technology",
year="2019",
publisher="Springer International Publishing",
pages="289--303"
}