meta(fuzzer): tracking issue for fuzzer improvements

[gakonst]

https://twitter.com/gakonst/status/1790770389523583163

Fuzz tests

Fuzzing-testing specific improvements, continuation of #4433

UX/Features

High

• feat: Add "after-all" hook for testing #4300
• feat: support time-based and continuous fuzzing and invariant testing  #990
• feat(cheatcodes): support native bound cheatcode #8788

Nice to have

• feat(cheatcodes): add ability to exclude certain custom errors and revert reason strings from failing tests #4271
• feat(fuzz): generate solidity regression tests from failures #8117
• Ability to get sorted arrays when fuzzing #4097
• feat: fuzz corpus saving and replay in standard format #2552 - change existing format to standard when available, see Allow Echidna & Medusa to share the same corpus crytic/medusa#234
• Console logs should ideally print _during_ fuzzing, not just after testing is complete #3844

Bugs

• bug(forge test): --fail-fast flag does not work as it is not applied across multiple test suites #6529
• Invalid Enum value when fuzzing #6623
• bug: state appears to be shared between tests when linked libraries are used #8639

Invariants

Invariant-testing specific improvements, continuation of #4438

UX/Features

High

• feat(forge): Add internal metrics capability #3607
• Allow forge's contract invariant testing to contribute to coverage #4007

Nice to have

• More granular control on invariant simulations #5018
• feat(invariant): extend the export of failed case to include traces as well #8114
• feat: vm.depth() cheatcode to return the depth of the current invariant run #2985
• Add invariant testing filter for excludeSelectors() #4352
• max_test_reject_rate: set a maximum test rejection rate per test function #4091
• feat: more flexible/powerful ways to define and test invariants #3452
• feat: test for reentrancy in invariant tests #1578

Performance

High

• feat(invariant): use storage layout to fuzz values from state by type  #8116
• feat(fuzz): do not populate dictionary with bytecode metadata #8115
• feat(forge): exclude precompiles by default in invariant tests #4287
• Built-in contracts like vm and the create2 factory should be excluded senders in invariants #4163
• feat: weight invariant selectors by number of selectors, not number of contracts. #2986

Benchmarks

High

• perf: fuzz/invariant benchmarks #3411
  • set up daily runner of https://github.com/grandizzy/fuzz-benchmarks/ + add more tests
• see details in feat: invariant benchmarks #7610
  • run Feedback on fuzzer benchmarking setup #4590
  • report as suggested in https://github.com/fuzz-evaluator/guidelines

Symbolic execution

• Symbolic Rust EVM #15

Mutation testing

• feat(testing): add mutation testing support #478

[zerosnacks]

this would be neat: #4000

[grandizzy]

this would be neat: #4000

yep, indeed! made a PR for it here #8338