How to performance-test a contract? Available tools?

[dckc]

It's required by the checklist:

• Mainnet 2 Deployment Checklist #8280

Perform performance and stress testing on the smart contract deployed on our testnet. Build test tools and scripts to simulate high loads, monitor resource usage, and identify bottlenecks. Document results, including performance metrics, failure points, and recommendations for improvement.

Any suggestions on how to do it? Are there any available tools?

cc @tgrecojs

[dckc]

Benchmarking Contract operations at the SwingSet VM Level

One somewhat mature set of tools for benchmarking smart contract operations runs within a SwingSet VM, but not using the cosmos-sdk consensus layer:

• The Benchmarkerator

There's a somewhat extended write-up:

• Running the benchmark for vault creation

[dckc]

end-to-end testing with transactions and queries

The ultimate test is to run an actual blockchain, submit transactions, and run queries.

For example, we have a shell script that tests minting a KREAd character:

KREAD_ITEM_OFFER=$(mktemp -t kreadItem.XXX)
node ./generate-kread-item-request.mjs > $KREAD_ITEM_OFFER
agops perf satisfaction --from $GOV1ADDR --executeOffer $KREAD_ITEM_OFFER --keyring-backend=test

agd query vstorage data published.wallet.$GOV1ADDR.current  -o json >& gov1.out
name=`jq '.value | fromjson | .values[2] | fromjson | .body[1:] | fromjson | .purses[1].balance.value.payload[0][0].name ' gov1.out`
test_val $name \"ephemeral_Ace\" "found KREAd character"

-- https://github.com/Agoric/agoric-3-proposals/blob/main/proposals/71%3Aupgrade-14/create-kread-item-test.sh

agops perf satisfaction is a test tool that submits a smart-wallet transaction and polls until the offer is satisfied.

Before KREAd went into production, we generated various levels of load using scripts like that and looked at various metrics. I think we ran them not just in an a3p context, but on an actual multi-validator-node test network in a kubernetes cluster, and we looked at the output using tools such as datadog. @toliaqat is there any more detail to share in this context?

[dckc]

end-to-end testing APIs

There are a few other tools for submitting transactions and making queries in the @agoric/synthetic-chain package.

See also:

• testing API to extend ava functional tests to end-to-end context #8963

... including recent multichain-testing work such as...

    const wdUser1 = await provisionSmartWallet(agoricAddr, {
      BLD: 100_000n,
      IST: 100_000n,
    });
    t.log(`provisioning agoric smart wallet for ${agoricAddr}`);

    const doOffer = makeDoOffer(wdUser1);

    const brands = await vstorageClient.queryData(
      'published.agoricNames.brand',
    );

-- https://github.com/Agoric/agoric-sdk/blob/master/multichain-testing/test/send-anywhere.test.ts

[erights]

Attn @gibson042

[dckc]

Here's a goal from internal discussion. I don't expect that this is self-explanatory, but I'll go ahead and share it...

• tool which submits txns at a fixed rate, and records completion time, and displays latency and acceptance rate, and some kind of "how busy is the chain" metric like block times or duration of the cosmic-swingset-begin-block -> cosmic-swingset-end-block-finish elapsed time
• run that at maybe 1txn/min for 10-15min, then increase to 2/min, then 3/min, etc, and plot the curves

we should learn how many computrons are spent to do one of those offers, to guess what our 65Mc limit will do to the scheduling.

Once we deploy to a multi-node net, we should see how it compares against the actual wallclock time needed by those validators.

---

p.s.

65Mc = 65,000,000 computrons

swingset has a run policy... cosmic-swingset uses a limit on computrons per block to make such a policy.

screenshot from swingset params:

[dckc]

Note @tgrecojs reports some interesting contract performance testing results for the Tribbles Airdrop contract on a local chain.

[dckc]

As discussed in today's office hours...

agoric-sdk/a3p-integration/proposals/z:acceptance/stakeBld.test.js

Line 37 in a4a4683

await agdWalletUtils.broadcastBridgeAction(GOV1ADDR, {

  await agdWalletUtils.broadcastBridgeAction(GOV1ADDR, {
    method: 'executeOffer',
    offer: {
      id: 'request-stake',
      invitationSpec: {
        source: 'agoricContract',
        instancePath: ['stakeBld'],
        callPipe: [['makeStakeBldInvitation']],
      },
      proposal: {
        give: {
          In: { brand: BLDBrand, value: 10n },
        },
      },
    },
  });

then something like...

  const istBalanceAfter = await retryUntilCondition(
    async () => getBalances([GOV1ADDR], 'uist'),
    // We only check gov1's IST balance is changed.
    // Because the reclaimed amount (0,015 IST) is less than execution fee (0,2 IST)
    // we might end up with less IST than the one before reclaiming the stuck payment.
    istBalance => istBalance !== istBalanceBefore,
    'tryExitOffer did not end up changing the IST balance',
    { log: t.log, setTimeout, retryIntervalMs: 5000, maxRetries: 15 },
  );

see also

• testing API to extend ava functional tests to end-to-end context #8963

[dckc]

In qualifying Fast USDC for release, we (@turadg , @mhofman , and co) came up with some guidelines to recommend as well as some techniques for addressing them.
I'm saving this in draft form; stay tuned for refinements.

We distinguished between "resource consumption" testing and "divergence" testing. The latter (e.g. #10940 for Fast USDC) needs coverage whereas the former (#10890) needs realistic production loads.

Resource consumption testing

While this often goes by “perf” testing, MN2 qualification is less about user-perceived latency and throughput and more about impact on long-term health of the chain, including leaked resources as well as consuming too much compute.

Structure

• Set up an “iteration” which will be repeated; all product functions of the contract SHOULD be covered
• Capture all the metrics for each iteration

Per iteration

• Exported objects
  • no growth, or growth corresponding to product features
  • [int] clistEntries
  • [int] kernelObjects
  • In boot tests, available on controller.getStats()
• Pending work
  • No pending work after operation is complete
  • [int] promiseQueueLength
  • [int] kernelPromises
  • In boot tests, available on controller.getStats()
• Vstorage
  • [int] Count of entries
  • [int] Total size
    • len(serialize(mock map))
  • Treating append-mode as set-mode, which is what RunUtils vstorage mock does
• Computrons
  • Compute is consistent per iteration
  • [int] Computrons
• Heap
  • [int] uncompressedSize of ‘current’ (i.e. latest) snapshot of each vat
    • Available from snapStore.listAllSnapshots()

While local node.js workers can be used for the first few categories for better DX (speed, debugger support...), Computrons and Heap can only be measured on XS workers.

Divergence testing

Just needs coverage to exercise all the code paths and confirm no divergence happens (with >1 validators so a3p and current multichain-testing are insufficient).

The report is to describe the coverage and show that no divergence occurred.

[mhofman]

The divergence testing in this case was required because Fast USDC is the first deployed contract exercising new cosmos features of the chain. Our JS environment is fully deterministic. golang however is specified to be non deterministic in a bunch of cases (and explicitly implements variations for non deterministic behavior). I do not expect any other contract to have to go through any requirement to test for divergence/non-determinism.