Test Command (a.k.a Dredd-lite)

[barrett-schonefeld]

User story
With one command, I’d like to validate status codes and response schemas (from an actual server implementation) against all the endpoints and operations in an OpenAPI description.

Problem
To test the consistency of a server implementation with its API description (using the validation proxy), users must decide which requests to make on a per-API basis and write individual scripts. It would be nice to provide a suite of requests to send through the validation proxy based on the API description.

Solution

To achieve this, we need to parse the OpenAPI description, gather the endpoints and operations, and make requests (both valid and invalid) to those endpoints with different parameters, request bodies, etc. through the validation proxy.

Through this we will, with one command, validate an API against its OpenAPI description and determine if the server-side implementation and/or the OpenAPI description need to be updated.

My team is currently researching tools that may be helpful in pursuit of this goal, and we may want to contribute this to Prism over the next few months. Is the Prism community interested in this feature, and will it accept our contribution?

Follow-up
This feature would be much improved (for my team’s purposes) if the validation proxy does not log errors for requests. We want the validation proxy to pass the request to the server as-is and validate the response. Would it be simple to disable request validation in the validation proxy?

[XVincentX]

Hey,

thanks for the message, super appreciated. This is a feature that has been popping up now and then by our users, which is a little bit what Dredd is doing (throwing request at your entire specification looking for inconsistencies).

I am not opposed to the feature (this is actually something that Prism v2 supports with the help of Stoplight's platform through Scenarios) but it must planned carefully (gathering the requirements, understanding the use cases).

Just to give you an example, here are some of the points that we should think about:

• How should Prism go through the operations?
  • By the way they appear in the document?
  • What if the order is not representative of the real operation order?
  • Shall we use a configuration file?
• Should Prism store some intermediate state to use in the successive calls, or shall we assume that the server will figure out how to respond correctly to calls that might not make sense?
  • Maybe Prism should be programmable with an hook model like the same that Dredd is currently using?

These are just some of the things that need to be sorted out, but likely there are way more.

I think the first step would be for you to write your requirements, what are you trying to solve, and go from there.

For the short term, I really think you should check out Dredd which fulfils your use case pretty completely

Cheers!

[barrett-schonefeld]

Vincent,

I appreciate the quick and thorough response!

Dredd is something we are investigating as well. Dredd seems to align directly with our use case but may require a similar amount of effort to provide OpenAPI 3 support.

I am leaning toward Prism because of the excellent organization and documentation of the project, the interesting opportunity for live request validation, and communication (for example, your response on this issue).

We will take some time to address the points you raised and provide another response here.

[XVincentX]

but may require a similar amount of effort to provide OpenAPI 3 support.

Dredd supports OpenAPI 3.0! Although the original author has left Apiary/Oracle (I used to work there, by the way) I still think it has a decent level of support and at least bug fixes. I'd give it a try.

I am leaning toward Prism because of the excellent organization and documentation of the project, the interesting opportunity for live request validation, and communication (for example, your response on this issue).

Aha thanks, I appreciate. We try to do our best.

[mimkorn]

I was also thinking of aiming Dredd at a Prism Validation Proxy to cover anything Dredd wouldn't catch due to its OpenAPI 3 parser being a bit slower to catch up: https://github.com/apiaryio/api-elements.js/blob/master/packages/openapi3-parser/STATUS.md Using Schemathesis would be another alternative, though when I tried that, hitting it via Prism ended up in this bug #1064

[philsturgeon]

This is something we've discussed, as removing the need for people to need to create scenarios just for basic contract testing was part of the plan for v3.x, which we've started with contract testing and hope to continue with a few other options.

In the mean time, using the proxy with a basic script which makes a bunch of requests to your API via the proxy will get the job done in many ways.

We'll need to plan this out properly internally, so leave it with us.

If anyone is looking at this thinking "I too would like to see this" I'd ask you to use the 👍 button up top to avoid bumping it, as we'll get to it when our roadmap cools off a little. We've got some huuuuuuge stuff on our plate for the next month or two. 🥴

[philsturgeon]

I've spent a lot of time thinking about this one. I'm glad we focused on making mocking and proxying the most effective we could, and now it's even possible to mix the two sensibly #1426. The goal has been to provide all sorts of mocking, proxying and contract testing in ways that don't just make prism feel like a random HTTP toolbox of awkwardly lumped together functionality.

With that progress made, we could consider having a test runner in Prism under a prism test {doc} {upstream}, which constructs requests from the http-spec request, fires them off to the upstream API using Prism HTTP Client like the proxy does, then spits out errors when it finds them much like the SL-Violations currently does. This would basically be Dredd minus their "hooks" functionality (which could certainly be figured out later).

I see it come up now and then like in this thread https://apisyouwonthate.slack.com/archives/C6EEULS87/p1599554457100700 where people are looking to have a brand new test suite that they don't have to spend much time making. I personally prefer to mix contract testing into the existing test suite when there is one, but I do understand the use case of improving a legacy API, having some sort of test runner to go through the docs, create example requests, and see what happens is at least some kind of testing, because more than just contract violations will appear, maybe some 500's escape, syntax errors, or even authentication errors!

Even if it's just a stepping stone towards eventually getting that codebase under control and adding a real test suite, having a test runner you don't have to create, or "keep in sync" is pretty handy in certain situations.

Is there anyone out there in open-source land who'd be interested in taking this on? We can work out a tree related bribe no doubt.