-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[JVM] Add retry logic for no coverage gain (#742)
This PR relocate the calculation of coverage total and coverage diff into the checking and retry loop. This fixes can then use the lively caluclated coverage diff to determine if the successfully build and run harness increase the project coverage or not. If it does not increase the project coverage, use the new retry logic prompts to ask LLM to help fixing the harness. The fixing of errors in generated harness as well as harness with no coverage increase are count together. After this fix, it increase the rate of successfully build and run for generated harness and make sure more of them could have help improveing project coverage. *Remark, this coverage feedback approach is currently only work exclusively for JVM projects.* --------- Signed-off-by: Arthur Chan <[email protected]>
- Loading branch information
1 parent
1eb15f3
commit fe43b0c
Showing
6 changed files
with
199 additions
and
54 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| I'm a security engineer looking to write good fuzzing harnesses. I want you help me improve my fuzzing harness so it could covers more part of the code. | ||
|
|
||
| The target library is {TARGET_REPO}. | ||
|
|
||
| The target project is implemented in the Java programming language; therefore, the harness should also be written in Java. | ||
| The fuzzing harness must be executable within the Jazzer fuzzing framework. | ||
|
|
||
| Below is the source code of the target fuzzing harness that I would like to improve: | ||
| <code> | ||
| {GENERATED_HARNESS} | ||
| </code> | ||
|
|
||
| For reference, the source code for all existing harnesses of the project is provided below, separated by `---`: | ||
| <code> | ||
| {EXISTING_HARNESS} | ||
| </code> | ||
|
|
||
| Additionally, a list of all public methods and constructors of the project is included for your reference, you should try to expand the fuzzing harness that calls these targets to improve the overall fuzzing coverage: | ||
| {PUBLIC_METHODS} | ||
|
|
||
| Your task is to improve the target fuzzing harness provided above to increase code coverage for additional parts of the project that are not covered by the existing fuzzing harnesses. Please ensure that the changes made are minimal. | ||
| In your response, include ONLY the code for the harness, nothing more. You should wrap the code in <code></code> tags. | ||
|
|
||
| Here is an additional list of requirements that you MUST follow. | ||
| <requirements> | ||
| <item>NEVER use any methods from the <code>java.lang.Random</code> class in the generated code.</item> | ||
| <item>NEVER use any classes or methods in the <code>java.lang.reflect</code> package in the generated code.</item> | ||
| <item>NEVER use the @FuzzTest annotation for specifying the fuzzing method.</item> | ||
| <item>NEVER use any assert, printing and logging statements in the generated harness.</item> | ||
| <item>NEVER use any multithreading or multi-processing approach.</item> | ||
| <item>You MUST create the object before calling the target method.</item> | ||
| <item>Please use {HARNESS_NAME} as the Java class name.</item> | ||
| <item>You MUST invoke the close method of any resource class objects that implements the java.lang.AutoCloseable interface in the finally block after the target method is invoked.</item> | ||
| <item>Always create the fuzzing harness from the following templates: | ||
| <code> | ||
| import com.code_intelligence.jazzer.api.FuzzedDataProvider; | ||
| // Other imports | ||
|
|
||
| public class {HARNESS_NAME} { | ||
| public static void fuzzerInitialize() { | ||
| // Initializing objects for fuzzing | ||
| } | ||
|
|
||
| public static void fuzzerTearDown() { | ||
| // Tear down objects after fuzzing | ||
| } | ||
|
|
||
| public static void fuzzerTestOneInput(FuzzedDataProvider data) { | ||
| // Use the FuzzedDataProvider object to generate random data for fuzzing | ||
|
|
||
| // Fuzz by invoking the target method with random parameters / objects generated above. | ||
| } | ||
| } | ||
| </code></item> | ||
| <item> | ||
| You MUST ONLY use any of the following methods from the FuzzedDataProvider of the Jazzer framework for generating random data for fuzzing. | ||
| If the needed return value is not found in the table, try use constructors or methods to create the needed random object. But you MUST try your best to randomise the random object with the methods in the table. | ||
|
|
||
| | Method | Return Value | | ||
| |---------------------------------------------|---------------------------------------| | ||
| | `consumeBytes(int length)` | `byte[]` | | ||
| | `consumeRemainingAsBytes()` | `byte[]` | | ||
| | `consumeString(int length)` | `String` | | ||
| | `consumeRemainingAsString()` | `String` | | ||
| | `consumeBoolean()` | `boolean` | | ||
| | `consumeInt(int min, int max)` | `int` | | ||
| | `consumeInt()` | `int` | | ||
| | `consumeLong(long min, long max)` | `long` | | ||
| | `consumeLong()` | `long` | | ||
| | `consumeFloat(float min, float max)` | `float` | | ||
| | `consumeFloat()` | `float` | | ||
| | `consumeDouble(double min, double max)` | `double` | | ||
| | `consumeDouble()` | `double` | | ||
| | `consumeChar()` | `char` | | ||
| | `consumeChar(char min, char max)` | `char` | | ||
| | `consumeShort(short min, short max)` | `short` | | ||
| | `consumeShort()` | `short` | | ||
| | `consumeRemainingAsCharSequence()` | `CharSequence` | | ||
| | `consumeBytestring()` | `byte[]` | | ||
| | `consumeBigInteger(int minNumBits)` | `BigInteger` | | ||
| | `consumeEnum(Class<E> enumType)` | `E` (Enum type) | | ||
| | `consumeProbabilityDouble()` | `double` | | ||
| | `consumeFraction()` | `double` | | ||
| | `pickValue(T... values)` | `T` (Type of value) | | ||
| | `pickValue(List<T> values)` | `T` (Type of value) | | ||
| | `consumeByte()` | `byte` | | ||
| | `consumeIntList(int length)` | `List<Integer>` | | ||
| | `consumeLongList(int length)` | `List<Long>` | | ||
| | `consumeFloatList(int length)` | `List<Float>` | | ||
| | `consumeDoubleList(int length)` | `List<Double>` | | ||
| | `consumeCharList(int length)` | `List<Character>` | | ||
|
|
||
| </item> | ||
| </requirements> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters