OpenApi based response checks

[LennSar]

Hi there,

I wanted to touch on this quote from the documentation:

That leaves part #4: defining the expected responses. But that's the part of the test program you have to fill in yourself. Tcases for OpenAPI has no way to predict what your API will do for any request, much less a request using random inputs.

I'm pretty sure you know best yourself but I have to disagree on this. We took a lot of effort to write an OpenApi.yaml which covers a lot of information on how the reponse should look like. Of course you cannot tell the exact answer but you could check whether the returned values are in the specified range, according to the defined regex or part of the defined enum. All this could be added during the automatic test case generation based on the OpenApi documentation. SmartBear ReadyApi for example does exactly that.

So my question would be: Is tcases already capable of doing something like that and I just didn't figure out how to do that yet and if that's not the case: Is this something you think about adding in the future?

Oh and of course I need to say: Thank you so much to take the time and effort to write this tool and make it public. As a Java novice I had a real hard time to get it running (and I would have failed if it weren't for the help of a java experiences colleague) but now where it does I realised that this a crazy powerful tool that at the end is much better than many programs you have to pay for.

Thanks!

Lennart

[kerrykimbrough]

Hi, Lennart

You make some very good points. Response validation is certainly a weak spot in Tcases for OpenAPI.

To answer your questions... Can the current version of TFOA generate code that checks response schema conformance? No, it can't. Could this be added to a future version of TFOA? Yes, I believe that is technically possible, with certain limitations.

One issue is that, in general, response data can be encoded according to many different media types. application/json is a common type, but it is not the only possibility. It's not practical for TFOA to support every possible media type. For some media types, it might not even be possible at all. On the other hand, support for application/json might be sufficient for many users. Which media types does your API use?

For response data that uses the application/json media type, it's possible for TFOA to generate the kind of response validation code that you suggest. I will definitely consider adding this feature to the next version of TFOA.

[LennSar]

Hi Kerry,

thanks for the quick reply. The issue of response data encoding sure does dramatically increase the amount of effort required. Even if it's only done for application/json (which fortunately we do use as well) it wouldn't be a small update to get all the informations required from the OpenApi Documentation. In our case we are using a lot of references which would further increase the required effort.
I'm really happy to read you're consider adding it in the next version of TFOA. If I can be of any help like testing things on our API, please let me know. I'll be happy with what I have so far and stay posted on any updates.

Thanks,

Lennart

[kerrykimbrough]

@LennSar Response validation is here! See release 3.8.0.

[LennSar]

Hi Kerry, thanks for the exciting news. You are fast! Really excited to take a look on the update! Thanks, Lennart From: Kerry Kimbrough ***@***.***> Sent: Thursday, April 14, 2022 7:56 PM To: Cornutum/tcases ***@***.***> Cc: Gardian, Lennart ***@***.***>; Mention ***@***.***> Subject: Re: [Cornutum/tcases] OpenApi based response checks (Discussion #219) @LennSar<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLennSar&data=04%7C01%7Clennart.gardian%40sartorius.com%7C2e05eef8d85345fc02e108da1e4009ed%7C8c7a02b0b4094a1483326772575f453b%7C0%7C0%7C637855557637757473%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RHV%2F3zJslitZrA%2B%2B%2Bqsj8ZPZ7%2FLG5HZf0JPjJxSk%2FeU%3D&reserved=0> Response validation is here! See release 3.8.0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FCornutum%2Ftcases%2Fblob%2Fmaster%2FReleaseNotes.md%23380&data=04%7C01%7Clennart.gardian%40sartorius.com%7C2e05eef8d85345fc02e108da1e4009ed%7C8c7a02b0b4094a1483326772575f453b%7C0%7C0%7C637855557637757473%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=D6WizRUoFuzlEdZ3lUKy8Irh4Jx%2BLsD7frItDSyHbrw%3D&reserved=0>. - Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FCornutum%2Ftcases%2Fdiscussions%2F219%23discussioncomment-2568693&data=04%7C01%7Clennart.gardian%40sartorius.com%7C2e05eef8d85345fc02e108da1e4009ed%7C8c7a02b0b4094a1483326772575f453b%7C0%7C0%7C637855557637757473%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=LgNnQqcttFsQaXX1hKMdfJqMNRdvu7HhD3IL%2ByIbfrA%3D&reserved=0>, or unsubscribe<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAYG3AQHI32HE3W6PONKHJADVFBL3BANCNFSM5QVDOVTA&data=04%7C01%7Clennart.gardian%40sartorius.com%7C2e05eef8d85345fc02e108da1e4009ed%7C8c7a02b0b4094a1483326772575f453b%7C0%7C0%7C637855557637757473%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iXwJjwFotCiOlzKObVdRvxww1c%2BwKfr%2BoxXMtaoX%2Bl8%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>