op-e2e

op-e2e

Issues: monorepo

Pull requests: monorepo

Design docs:

• [test infra draft design-doc]: active discussion of end-to-end testing approach

test infra draft design-doc

op-e2e is a collection of Go integration tests. It is named e2e after end-to-end testing, for those tests where we integration-test the full system, rather than only specific services.

Quickstart

make test-actions
make test-ws

Overview

op-e2e can be categorized as following:

• op-e2e/actions/: imperative test style, more DSL-like, with a focus on the state-transition parts of services. Parallel processing is actively avoided, and a mock clock is used.
  • op-e2e/actions/*: sub-packages categorize specific domains to test.
  • op-e2e/actions/interop: notable sub-package, where multiple L2s are attached together, for integration-testing across multiple L2 chains.
  • op-e2e/actions/proofs: notable sub-package, where proof-related state-transition testing is implemented, with experimental support to cover alternative proof implementations.
• op-e2e/system: integration tests with a L1 miner and a L2 with sequencer, verifier, batcher and proposer. These tests do run each service almost fully, including parallel background jobs and real system clock. These tests focus less on the onchain state-transition aspects, and more on the offchain integration aspects.
  • op-e2e/faultproofs: system tests with fault-proofs stack attached
  • op-e2e/interop: system tests with a distinct Interop "SuperSystem", to run multiple L2 chains.
• op-e2e/opgeth: integration tests between test-mocks and op-geth execution-engine.
  • also includes upgrade-tests to ensure testing of op-stack Go components around a network upgrade.

action-tests

Action tests are set up in a compositional way: each service is instantiated as actor, and tests can choose to run just the relevant set of actors. E.g. a test about data-availability can instantiate the batcher, but omit the proposer.

One action, across all services, runs at a time. No live background processing or system clock affects the actors: this enables individual actions to be deterministic and reproducible.

With this synchronous processing, action-test can reliably navigate towards these otherwise hard-to-reach edge-cases, and ensure the state-transition of service, and the interactions between this state, are covered.

Action-tests do not cover background processes or peripherals. E.g. P2P, CLI usage, and dynamic block building are not covered.

system-tests

System tests are more complete than action tests, but also require a live system. This trade-off enables coverage of most of each Go service, at the cost of making navigation to cover the known edge-cases less reliable and reproducible. This test-type is thus used primarily for testing of the offchain service aspects.

By running a more full system, test-runners also run into resource-limits more quickly. This may result in lag or even stalled services. Improvements, as described in the [test infra draft design-doc], are in active development, to make test execution more reliable.

op-e2e/opgeth

Integration-testing with op-geth, to cover engine behavior, without setting up a full test environment. These tests are limited in scope, and may be changed at a later stage, to support alternative EL implementations.

Product

Optimization target

Historically op-e2e has been optimized for test-coverage of the Go OP-Stack. This is changing with the advance of alternative OP-Stack client implementations.

New test framework improvements should optimize for multi-client testing.

Vision

Generally, design-discussion and feedback from active test users converges on:

• a need to share test-resources, to host more tests while reducing overhead.
• a need for a DSL, to better express common test constructs.
• less involved test pre-requisites: the environment should be light and simple, welcoming new contributors. E.g. no undocumented one-off makefile prerequisites.

Design principles

• Interfaces first. We should not hardcode test-utilities against any specific client implementation, this makes a test less parameterizable and less cross-client portable.
• Abstract setup to make it the default to reduce resource usage. E.g. RPC transports can run in-process, and avoid websocket or HTTP costs, and ideally the test-writer does not have to think about the difference.
• Avoid one-off test chain-configurations. Tests with more realistic parameters are more comparable to production, and easier consolidated onto shared testing resources.
• Write helpers and DSL utilities, avoid re-implementing common testing steps. The better the test environment, the more inviting it is for someone new to help improve test coverage.
• Use the right test-type. Do not spawn a full system for something of very limited scope, e.g. when it fits better in a unit-test.