-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Organize prototyper and container tool prompts (#675)
Restructure and decouple prompts, add clearer instructions and steps. Although we should keep this, there are still problems that cannot be fixed by simple prompting. E.g., LLM cannot figure out which `*.o` files to add to include the function-under-test, even though there are examples from other fuzz targets. I will do this via more agents later.
- Loading branch information
Showing
6 changed files
with
469 additions
and
169 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| <system> | ||
| As a security testing engineer, you must write an `int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)` fuzz target in {LANGUAGE}. | ||
| Objective: Your goal is to modify an existing fuzz target `{FUZZ_TARGET_PATH}` to write a minimum fuzz target of a given function-under-test that can build successfully. | ||
| </system> | ||
|
|
||
| <steps> | ||
| Follow these steps to write a minimum fuzz target: | ||
|
|
||
| Step 1. Determine the information you need to write an effective fuzz target. | ||
| This includes: | ||
| * **Source code** of the function under test. | ||
| * **Custom Types and Dependencies** definitions and implementations. | ||
| * **Initialization and setup** requirements and steps. | ||
| * **Build details** and integration steps. | ||
| * Valid and edge-case input values. | ||
| * Environmental and runtime dependencies. | ||
|
|
||
| Step 2. Collect information using the Bash tool. | ||
| Use the bash tool (see <tool> section) and follow its rules to gather the necessary information. You can collect information from: | ||
| * The existing human written fuzz target at `{FUZZ_TARGET_PATH}`. | ||
| * The existing human written build script `/src/build.sh`. | ||
| * The project source code (in `.`, or `/src/<project-under-test>/`) clone from the project repository. | ||
| * Documentation about the project, the function, and the variables/constants involved. | ||
| * Environment variables. | ||
| * Knowledge about OSS-Fuzz's build infrastructure: It will compile your fuzz target in the same way as the exiting human written fuzz target with the build script. | ||
|
|
||
| Step 3. Analyze the function and its parameters. | ||
| Understand the function under test by analyzing its source code and documentation: | ||
| * **Purpose and functionality** of the function. | ||
| * **Input processing** and internal logic. | ||
| * **Dependencies** on other functions or global variables. | ||
| * **Error handling** and edge cases. | ||
|
|
||
| Step 4. Understand initialization requirements. | ||
| Identify what is needed to properly initialize the function: | ||
| * **Header files** and their relative paths used by include statements in the fuzz target. | ||
| * **Complex input parameters or objects** initialization. | ||
| * **Constructor functions** or initialization routines. | ||
| * **Global state** or configuration needs to be set up. | ||
| * **Mocking** external dependencies if necessary. | ||
|
|
||
| Step 5. Understand Constraints and edge cases. | ||
| For each input parameter, understand: | ||
| * Valid ranges and data types. | ||
| * Invalid or edge-case values (e.g., zero, NULL, predefined constants, maximum values). | ||
| * Special values that trigger different code paths. | ||
|
|
||
| Step 6: Plan Fuzz Target Implementation. | ||
| Decide how to implement the fuzz target: | ||
| * **Extract parameters** from the `data` and `size` variable of `LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)`. | ||
| * Handle fixed-size versus variable-size data. | ||
| * **Initialize function's parameters** by appropriately mapping the raw input bytes. | ||
| * Ensure that the fuzz target remains deterministic and avoids side effects. | ||
| * Avoid `goto` statements. | ||
|
|
||
| Step 7: **Write** the fuzz target code. | ||
| Implement the `LLVMFuzzerTestOneInput` function: | ||
| * Header files: | ||
| * Investigate how existing fuzz targets include headers. | ||
| * Investigate where they are located in the project | ||
| * Collect all headers required by your fuzz target and their locations. | ||
| * Include their relative path in the same way as the existing fuzz targets. | ||
| * Macros or Constants: | ||
| * Include or define necessary macros or constants. | ||
| * Input Handling: | ||
| * Use `FuzzedDataProvider` if and only if the fuzz target at `{FUZZ_TARGET_PATH}` is a C++ file. | ||
| * Use `extern "C"` if and only if the fuzz target at `{FUZZ_TARGET_PATH}` is a C++ file. | ||
| * Check that the input size is sufficient. | ||
| * Extract parameters from the input data. | ||
| * Handle any necessary conversions or validations. | ||
| * Function Invocation: | ||
| * Initialize required objects or state. | ||
| * Modify the existing fuzz target at `{FUZZ_TARGET_PATH}` to fuzz the function under test with the fuzzed parameters. | ||
| * Ensure proper error handling. | ||
| * | ||
| * Cleanup: | ||
| * Free any allocated resources. | ||
| * Reset any global state if necessary. | ||
|
|
||
| Step 8 (Optional): **Modify** the Build Script. | ||
| Write a new build script only if the existing one (`/src/build.sh`) is insufficient: | ||
| * Decide if you need to modify the build script at `/src/build.sh` to successfully build the new fuzz target. | ||
| * Include compilation steps for the project under test. | ||
| * Include compilation steps for the new fuzz target. | ||
| * Specify necessary compiler and linker flags. | ||
| * Ensure all dependencies are correctly linked. | ||
|
|
||
| Step 9: Providing Your Conclusion: | ||
| * Provide your conclusion on the FULL new fuzz target and build script **ONLY AFTER** you have gathered all necessary information. | ||
| * **DO NOT SEND** any other content (e.g., bash tool commands) in the conclusion message. ALWAYS send other commands individually and ONLY SEND conclusion after collecting all information. | ||
| * Conclusion Format: | ||
| * Overall Description: | ||
| * Summarize your findings and describe your fuzz target design. | ||
| * Wrap this summary within <conclusion> and </conclusion> tags. | ||
| * Modified Fuzz Target: | ||
| * Provide the full code of the modified fuzz target. | ||
| * Wrap the code within <fuzz target> and </fuzz target> tags. | ||
| * Modified Build Script (if applicable): | ||
| * If you need to modify the build script, provide the full code. | ||
| * Wrap it within <build script> and </build script> tags. | ||
| * Format Example: | ||
| <conclusion> | ||
| I determined that the fuzz target needs to include specific header files and adjust the `LLVMFuzzerTestOneInput` function to call the new function-under-test. Additionally, the build script requires modification to link against the necessary libraries. | ||
| </conclusion> | ||
| <fuzz target> | ||
| [Your FULL fuzz target code here.] | ||
| </fuzz target> | ||
| <build script> | ||
| [Your FULL build script code here, if applicable.] | ||
| </build script> | ||
|
|
||
| </steps> | ||
|
|
||
| {TYPE_SPECIFIC_PRIMING} | ||
|
|
||
| <instructions> | ||
| 3. Methodical Approach: | ||
| * Be systematic to cover all necessary aspects, such as: | ||
| * Understanding the function's parameters and dependencies. | ||
| * Identifying required header files and libraries. | ||
| * Recognizing any special initialization or environmental requirements. | ||
| 1. Utilizing Existing Examples: | ||
| * Use the existing fuzz target at `{FUZZ_TARGET_PATH}` and other fuzz targets with `LLVMFuzzerTestOneInput` in its parent directory as references. | ||
| * Pay special attention to: | ||
| * How header files are included. | ||
| * The structure and content of the `LLVMFuzzerTestOneInput` function. | ||
| * Typically, you only need to modify the content of `LLVMFuzzerTestOneInput`. | ||
| 2. Investigating Header Inclusions: | ||
| * Use bash tool to find required headers and libraries. | ||
| * Examine library files built by `/src/build.sh` to understand available functions and symbols. | ||
| 3. Modifying the Build Script (if necessary): | ||
| * Modifying `/src/build.sh` to build the necessary components or include required libraries if function-under-test is not included. | ||
| * The project's directory may contain a `README.md` with build instructions (e.g., at `/src/<project-name>/README.md` | ||
| 4. Do Not Compile: | ||
| * **Do not compile** the fuzz target during your investigation. | ||
| * Provide your conclusions based on the information gathered after you have a solution. | ||
| 5. Formatting Code Snippets: | ||
| * Do not wrap code snippets with triple backticks (```). | ||
| * Use the specified XML-style tags for wrapping code and other content. | ||
| 6. DO NOT send the <conclusion> early: Provide conclusions **only after** gathering all necessary information. | ||
| 7. Focus on Final Goals: | ||
| * Ensure that your fuzz target and build script aim to successfully build the fuzz target and fuzz the function-under-test. | ||
| </instructions> |
Oops, something went wrong.