Hard to reproduce tests

[zmrocze]

Severity	CVSS	Vulnerability Type
None	0.0	insufficient-tests

Description

Tests that can be executed by the developers easily and reproducibly are a requirement for reliable software development. There are a couple of issues complicating the execution of the test suite.

Execution against preprod

The tests are executed on the preprod chain. This approach gives the best idea of how the protocol behaves in the wild. However, it is challenging to reproduce the tests, which for a developer is helpful to do often, even as often as on every validator change.

The protocol initialization mints a one-shot-mint NFT token by spending a specified UTxO. This means that on every NFT script change a new UTxO needs to be queried on the blockchain, then hardcoded into the validator's source code, the new validators be compiled and only then the test suite can run. When the NFT is not changed, then the mintNFT action can be commented out. In other words, many source code modifications need to be performed to run the tests with updated policies.

Fully applied scripts

This issue connects with the above one. The Plutus scripts get fully parameterized to yield the validator and minting policies already in the on-chain code. As all the 3 scripts depend directly or indirectly on the UTxO to be spent in the one-shot-mint transaction, the source code needs to be recompiled with queried information every time an NFT policy changes. This also forbids test execution in an emulator.

Unknown dependency versions

The MLabs team could not reproduce the build of the typescript off-chain tests with the provided package versions. The team was able to build the test suite using the newer nodejs version (v20.11.1). The node packages used as project dependencies (so ts-node-esm, typescript) should be specified and their versions pinned in the tests/package.json file.

Recommendation

There exist many tools for testing Plutus scripts, Plutip, PSM to name some open-source ones developed by MLabs. Every tool covers slightly different space and has its trade-offs, therefore the codebase would benefit from integrating both.

Here MLabs recommends mirroring the existing test suite in the Lucid emulator and provides a contribution implementing such a change. This approach has been chosen because it integrates easily into the existing codebase and is the most feasible to implement in the given time span. The suggested implementation is provided without guarantee of any kind and the SingularityNet team is advised to review it thoroughly. With the added ergonomics of running the test suite completely off-line and reproducibly, there was introduced the complexity of applying the validator scripts dynamically in the off-chain code. It should be noted that sooner or later the ability to apply scripts dynamically becomes a necessity of the off-chain application utilizing the validators.

The PR 14 brings changes to the scripts allowing the off-chain to parameterize them dynamically and defines a parallel test suite that runs against an emulator instead of the preprod network. The implementation details are discussed in the PR.