Eth 2.0 API Conformance Tester MVP

[skmgoldin]

Background

Efforts to formally specify an Ethereum 2.0 API are underway but will not be near complete for the upcoming multi-client testnet. For the near-term, we have decided that we will expose the Lighthouse/Teku API for our testnet offering while the API specification work is ongoing.

While the Teku team has been working to implement an API matching that Lighthouse exposes, we have yet to rigorously assess how well these implementations cohere in practice. Furthermore, we will need a facility to assess the coherence of these APIs in an ongoing manner as the Lighthouse and Teku teams make new releases which we may desire to put into production.

To meet our need for reproducible assessments of API coherence between clients, we will build a core component of the automated Eth 2.0 API conformance certification system described in the Ethereum 2.0 API Strategy proposal: the test runner!

Eth 2.0 API Conformance Tester MVP

The MVP is a CLI tool wrapping a library for testing the ability of a URL to provide conformant responses to queries generated from OpenAPI definitions. The tool will be able to output test results as JSON, or in a human-readable format. It will return a 0 exit code if all tests pass successfully, and non-zero otherwise.

$ eth2conf localhost:5051
Running tests...
Done!
 API   | PASS
---------------
 API_0 | ✅
 API_1 | ✅
 API_2 | ❌
 API_3 | ✅

ERRORS
---------------
API_2
Could not parse expected type "SignedBeaconBlock" in response.
Response:
<...>

This tool will not manage the provisioning of clients to test against.

Supporting technology: https://github.com/getkin/kin-openapi (OAPI data validator)

Note: what spec are we testing against?

Because we have decided that we will expose the Lighthouse/Teku API while the broader community API specification work is ongoing, we will use Teku's generated (and release-tagged) OpenAPI schema as our source of truth. Because the eth2.0-APIs schema is also OpenAPI, eventually cutting over should be as simple as changing a URL and updating relevant test cases.

[ryanschneider]

This all sounds good to me! BTW if any discussion threads on this issue end up going long, it might be easier to PR this as a proposal .md inside private-wiki to solicit feedback.

[ncocchiaro]

Sounds good, do you see anything we can leverage, reuse or extend from Versus or Ethspam here?

[egalano]

Yup sounds good to me too. +1 to what Nicola said. Take a look at the work Andrey did in Versus for comparing responses. It doesn't do anything with a declarative spec for responses at the moment though.

[timmyers]

LGTM. At what point do you think we should open source this so client teams have the ability to use it? Should we get that process started with legal?

[egalano]

That's usually a less than one week process, sometimes just a two email exchange. Once we know what license we'd use and can describe the business reasons for open-sourcing we can email legal

[skmgoldin]

it might be easier to PR this as a proposal .md inside private-wiki to solicit feedback.

Since this was already approved as part of the Ethereum 2.0 API Strategy, I don't think it needs to go through that level of review again. Just wanted to get a little feedback on this slightly more detailed view since I estimate it'll be a good-sized chunk of work.

do you see anything we can leverage, reuse or extend from Versus or Ethspam here?

I will take a look and report back!

At what point do you think we should open source this so client teams have the ability to use it? Should we get that process started with legal?

From my perspective the only blocking thing would be supporting the eth2.0-api repo spec versus the Teku one. We could support both simultaneously and eventually deprecate the Teku one. If that ends up being not too much work then I can get it into the initial release. It depends on if Teku today faithfully implements the existing eth2.0-APIs subset in its API, in which case the test cases should be reusable.

[skmgoldin]

I looked at Versus and Ethspam. I don't know that I can directly reuse/extend anything from either, but I am definitely inspired by the architecture for RPC generation in Ethspam and might use a similar approach to generate some REST requests in this repo.

Last week I was researching different approaches to take for test generation here. I was initially intrigued by the idea of generating a comprehensive test suite programmatically from the Swagger spec itself. But it turns out this is harder to do than you might think, and I wasn't able to find any existing tools to do it that seemed well-maintained. There's a Microsoft research paper on this specific topic: https://www.microsoft.com/en-us/research/uploads/prod/2018/04/restler.pdf

In the meantime I’ve settled on just hand-building tests within an architecture that will allow for generative testing in the future if somebody in the community feels it would be valuable to have. We could initially do Ethspam-style generative requests which wouldn't be as comprehensive as the REST-ler approach, but could still be valuable.

[skmgoldin]

This MVP is done! The only thing I didn't implement was #6, but I can do it quickly after vacation. It doesn't need to block open-sourcing this.