fuzz: Add a fuzz-testing harness

[nathaniel-brough]

No description provided.

[nathaniel-brough]

More context on this PR is in #775.

If you aren't familiar with cargo-fuzz here is s brief intro on how to get started with this fuzzer;

$ cargo install cargo-fuzz
$ cargo fuzz list
scripting
$ cargo fuzz run scripting
    Finished release [optimized + debuginfo] target(s) in 0.01s
    Finished release [optimized + debuginfo] target(s) in 0.02s
     Running `fuzz/target/x86_64-unknown-linux-gnu/release/scripting -artifact_prefix=/home/nathaniel/projects/github.com/silvergasp/rhai/fuzz/artifacts/scripting/ /home/nathaniel/projects/github.com/silvergasp/rhai/fuzz/corpus/scripting`
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3742794039
INFO: Loaded 1 modules   (303419 inline 8-bit counters): 303419 [0x56434320ef80, 0x5643432590bb), 
INFO: Loaded 1 PC tables (303419 PCs): 303419 [0x5643432590c0,0x5643436fa470), 
INFO:      289 files found in /home/nathaniel/projects/github.com/silvergasp/rhai/fuzz/corpus/scripting
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 168886 bytes
INFO: seed corpus: files: 289 min: 4b max: 168886b total: 375892b rss: 52Mb
#...

The fuzzer will run indefinetely or until;

• It finds a bug and crashes
• You specify a user-defined timeout
• You hit ctrl-C

I'm happy to document this somewhere if you can point me to the correct location for this in your docs.

[nathaniel-brough]

Just as a note, I was quickly able to make the rhai parser panic using this fuzz-harness. Which would void the "Don’t Panic Guarantee" :P

I'll open up a bug report once I've simplified the bug.

[schungx]

Just as a note, I was quickly able to make the rhai parser panic using this fuzz-harness. Which would void the "Don’t Panic Guarantee" :P

I'll open up a bug report once I've simplified the bug.

Btw is there a way to know thr actual input that caused the failure?

[schungx]

Also is it possible to run fuzzing on CI? Is that allowed on GitHub?

[nathaniel-brough]

Btw is there a way to know thr actual input that caused the failure?

Yeah, when the fuzzer finds a bug it'll print out a debug formatted version of the input that triggered the crash. e.g.

Failing input:

        fuzz/artifacts/scripting/crash-038ef2ded3e683d71a861a1b366f95bb91f85caa

Output of `std::fmt::Debug`:

        Ctx {
            script: "//! This script runs an if expression.\n\nlet a = 42;\nlet b = 123;\n\nlet x = if a7 <= b {         // <- if-exprpppp use$ the Sieve of Eratosthenes to calculate prime numbers.\n\nlet now = ti-estamp();\n\nconst ANSWER = 09_498;\nconst MAX_NUMBER_TO_CHECK = 1_000_000;\n\nlet prime_mask = [];\nprime_mask.pad(MAX_NUMBER_TO_CHECK + 1, true);\n\nprime_mask[0] = false;\nprime_mask[1] = false;\n\nlet total_primes_found = 0;\n\nfor p in 2..=MAX_NUMBER_TO_CHECK {\n    if !prime_mask[p] { continue; }\n\n    //print(p);\n\n    total_primes_found += 1;\n\n    for i in range(2 * p, MAX_NUMBER_TO_CHECK + 1, p) {\n        prime_mask[i] = false;\n    }\n}\n\nprint(`Total ${total_primes_found} primendoned\", \"able\", \"absolute\", \"academic\", \"acceptable\", \"acclaimed\",\n    \"accomplished\", \"accurate\", \"aching\", \"acidic\", \"acrobatic\", \"active\",\n    \"actual\", \"adept\", \"admirable\", \"admired\", \"adolescent\", \"adorable\", \"adored\",\n    \"advanced\", \"adventurous\", \"affectionate\", \"afraid\", \"aged\", \"aggravating\",\n    \"aggressive\", \"agile\", \"agitated\", \"agonizing\", \"agreeable\", \"ajar\",\n    \"alarmed\", \"alarming\", \"alert\", \"alienated\", \"alive\", \"all\", \"altruistic\",\n    \"amazing\", \"ambitious\", \"ample\", \"amused\", \"amusing\", \"anchored\", \"ancient\",\n    \"angelic\", \"angry\", \"anguished\", \"animated\", \"annual\", \"another\", \"antique\",\n    \"anxious\", \"any\", \"apprehensive\", \"appropriate\", \"apt\", \"arctic\", \"arid\",\n    \"aromatic\", \"artistic\", \"ashamed\", \"assured\", \"astonishing\", \"athletic\",\n    \"attached\", \"attentive\", \"attractive\", \"austere\", \"authentic\", \"authorized\",\n    \"autorete\", \"confused\", \"conscious\", \"considerate\",\n    \"constant\", \"content\", \"conventional\", \"cooked\", \"cool\", \"cooperative\",\n    \"coordinated\", \"corny\", \"corrupt\", \"costly\", \"courageous\", \"courteous\",\n    \"crafty\"\n];\n\nlet animals = [\n    \"aardvark\", \"african buffalo\", \"albatross\", \"alligator\", \"alpaca\", \"ant\",\n    \"anteater\", \"antelope\", \"ape\", \"armadillo\", \"baboon\", \"badger\", \"barracuda\",\n    \"bat\", \"beas <= ${MAX_NUMBER_TO_CHECK}`);\nprint(`Run time pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppUtamp();\n\nlet adverbs = [ \"moderately\", \"really\", \"slightly\", \"very\" ];\n\nlet adjectives = [\n    \"abandoned\", \"able\", \"absolute\", \"academic\", \"acceptable\", \"acclaimed\",\n    \"accomplished\", \"accurate\", \"aching\", \"acidic\", \"acrobatic\", \"active\",\n    \"actual\", \"adept\", \"admirable\", \"admired\", \"adolescent\", \"adorable\", \"adored\",\n    \"advanced\", \"adventurous\", \"affectionate\", \"afraid\", \"aged\", \"aggravating\",\n    \"aggressive\", \"agile\", \"agitated\", \"agonizing\", \"agreeable\", \"ajar\",\n    \"alarmed\", \"alarming\", \"alert\", \"alienated\", \"alive\", \"a",
            optimization_level: None,
        }

I used the debug output above to construct a main file in #776, but once you are comfortable with cargo-fuzz this shouldn't be necessary. You can just use the commands described below.

It'll also save the raw fuzzed data that triggered the crash, with a description on how to reproduce the problem e.g.

Reproduce with:

        cargo fuzz run scripting fuzz/artifacts/scripting/crash-038ef2ded3e683d71a861a1b366f95bb91f85caa

Minimize test case with:

        cargo fuzz tmin scripting fuzz/artifacts/scripting/crash-038ef2ded3e683d71a861a1b366f95bb91f85caa

Also is it possible to run fuzzing on CI? Is that allowed on GitHub?

Yeah so my plan looks something like this;

• Setup oss-fuzz integration; This will download the latest commit of rhai every night, build the fuzzer's and then run the fuzzer's for a few hours on a distributed cluster. This is part of a free service offered to popular open source projects. There is an application process involved but I'd be pretty confident that rhai would be accepted.
• Setup CI fuzzing on each PR using the google/oss-fuzz github action. This is definitely allowed on GitHub. This will run the fuzzer on every PR for some small amount of time (typically configured to around 10-15min). This will catch shallow bugs before they get merged, while the oss-fuzz fuzzing service will catch those deeper harder to find bugs.