RESTler's body schema parameter fuzzing approach

[hidingturtle]

Description

Hello,

I wanted to ask about the current approach of how RESTler fuzzes the request bodies or rather its concrete workflow. In this paper it says this:

... function RENDER of Figure 3 generates all possible combinations of dictionary values for every request with several fuzzable type. For large dictionaries, this may result in astronomical numbers of combinations. In that case, a more scalable option is to randomly sample each dictionary for one (or a few) values, or to use combinatorial- testing algorithms [10] for covering, say, every dictionary value, or every pair of values, but not every k-tuple.

However, in another paper (with a new approach) it says that Schema Fuzzing Rules like "Drop", "Single" etc. are used. I can see this in the file restler/checkers/body_schema_fuzzer.py. However, it's implemented as a checker and checkers are usually applied later than the requests themselves as far as I understand. In addition, this checker is not a default checker I think.

So I wanted to ask how exactly parameter fuzzing works currently in RESTler since two approaches are listed in papers but it's not clear to me which exactly is used by default. I understand that for values a dictionary is used but it's not clear how the parameters are chosen.

[hidingturtle]

Ok, I see that the method with "Drop", "Single" etc. is actually used in the PayloadBodyChecker. However, the question still stands how this is handled by RESTler since checkers are applied afterwards. Does RESTler just try to generate valid sequences and then applies checkers like the PayloadBodyChecker?

[marina-p]

Hello @hidingturtle,

Correct, RESTler runs a "main algorithm" which has different strategies for what sequences of requests to generate, and orthogonally RESTler applies one or more checkers after each executed sequence (as applicable - not all checkers apply after every sequence). The checkers are designed to run independently from each other and not affect subsequent execution of the main algorithm (which may or may not be successful, depending on the service behavior / how well RESTler understands the service).

See generate_sequences in driver.py for the overall flow, and the render_one function in driver.py for what happens when executing a single sequence and where checkers are applied.

Thanks,

Marina

[marina-p]

RE: parameter fuzzing specifically - there are 3 different ways parameters may be fuzzed:

1. Fuzzing dictionary

These are either fuzzable_* values or custom payloads. RESTler will generate all combinations of these for a given request, up to max_combinations (default 20). This means that if there are many fuzzable parameters with several combinations each, by default only the first parameter will be fuzzed, since max_combinations will be reached before trying other values for the subsequent parameters. This is the approach described in this paper.

We recommend keeping the fuzzing dictionary small. For example, if the goal is to test many invalid values for the entire API (e.g. run through a large list of malformed data for some parameter type), it is recommended to use the more recently added invalid value checker (see 3) below), which will not exercise all combinations. There are currently no optimizations available in RESTler for dealing with very large combinations of valid parameters - increasing max_combinations to a large number is required to get the desired coverage for each parameter in a single RESTler run.

2. Payload Body Checker

This checker works as described in this paper. There is a specific recipe for both fuzzing the schema and payloads of json bodies only. These parameters are fuzzed independently, i.e. the payload body checker does not combine body fuzzing with query/path parameter fuzzing - this checker takes a request sequence and fuzzes the body of the last request only.

3. Invalid value checker

This is a newer checker that enables fuzzing each parameter independently with a specified set of values one by one. It has a separate dictionary and separate limit for the number of values to try.

Thanks,

Marina