Skip to content

Latest commit

 

History

History
139 lines (99 loc) · 5.21 KB

reproducing.md

File metadata and controls

139 lines (99 loc) · 5.21 KB
Raw

Reproducing libFuzzer and AFL crashes

*** note Requirements: For Windows, you must convert the forward slashes (/) to backslashes (\) in the commands below and use set command instead of export to set the environment variable (step 4). Note that these commands are intended to be used with cmd.exe, not PowerShell. Also, you may find these tips on how to debug an ASAN instrumented binary helpful.

[TOC]

Crashes reported as Reproducible

The majority of the bugs reported by ClusterFuzz have Reproducible label. That means there is a testcase that can be used to reliably reproduce the crash.

  1. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by ClusterFuzz, a link to it is next to "Reproducer testcase" in the bug description.

    For the rest of this walkthrough, we call the path of this file: $TESTCASE_PATH and the fuzz target you want to reproduce a crash on: $FUZZER_NAME (provided as "Fuzz Target" in the bug description).

  2. Generate gn build configuration:

gn args out/fuzz

This will open up an editor. Copy the gn configuration parameters from the values provided in GN Config section in the ClusterFuzz testcase report.

  1. Build the fuzzer:
autoninja -C out/fuzz $FUZZER_NAME
  1. Set the *SAN_OPTIONS environment variable as provided in the Crash Stacktrace section in the testcase report. Here is an example value of ASAN_OPTIONS that is similar to its value on ClusterFuzz:
export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0
  1. Run the fuzz target:
out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH

File a bug if you run into any issues.

Symbolizing stack traces

Stack traces from ASAN builds are not symbolized by default. However, you can symbolize them by piping the output into:

src/tools/valgrind/asan/asan_symbolize.py

Crashes reported as Unreproducible

ClusterFuzz generally does not report issues that it cannot reliably reproduce, unless the following condition is met. If a certain crash is occurring often enough, such a crash might be reported with Unreproducible label and an explicit clarification that there is no convenient way to reproduce it. There are two ways to work with such crashes.

  1. Try a speculative fix based on the stacktrace. Once the fix is landed, wait a couple days and then check Crash Statistics section on the ClusterFuzz testcase report page. If the fix works out, you will see that the crash is not happening anymore. If the crash does not occur again for a little while, ClusterFuzz will automatically close the issue as Verified.

  2. (libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is very similar to the one described above for the Reproducible crashes. The only differences are:

  • On step 1, instead of downloading a single testcase, you need to download corpus backup. This can be done using the following command:
gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip .
  • Alternatively, you can navigate to the following URL in your browser and download the latest.zip file:
https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME
  • Create an empty directory and unpack the corpus into it.
  • Follow steps 2-4 in the Reproducible section above.
  • On step 5, use the following command:
out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP
  • Wait and hope that the fuzzer will crash.

Waiting for a crash to occur may take some time (up to 1hr), but if it happens, you will be able to test the fix locally and/or somehow debug the issue.

Minimizing a crash input (optional)

ClusterFuzz does crash input minimization automatically, and a typical crash report has two testcases available for downloading:

  • An original testcase that has triggered the crash;
  • A minimized testcase that is smaller than the original but triggers the same crash.

If you would like to further minimize a testcase, run the fuzz target with the two additional arguments:

  • -minimize_crash=1
  • -exact_artifact_path=<output_filename_for_minimized_testcase>

The full command would be:

out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH

This might be useful for large testcases that make it hard to identify a root cause of a crash. You can leave the minimization running locally for a while (e.g. overnight) for better results.