Skip to content

introduce --fuzz=<limit> to make tests pass after a certain quantity of fuzz testing has completed #22900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andrewrk opened this issue Feb 15, 2025 · 0 comments
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. fuzzing
Milestone

Comments

@andrewrk
Copy link
Member

andrewrk commented Feb 15, 2025

There should be two modes of fuzzing, infinite mode, and finite mode. This issue is for finite mode.

Introduce --fuzz=<limit>, where the limit can be one of these:

  • time - a value such as 1s, 2m, 3h, 4d
  • iteration count - a value such as 100, 100K, 100M, 100G

Time-based fuzzing makes all fuzz tests finish successfully after the specified interval has passed. The time is global; adding more fuzz tests and keeping the time the same means less time spent on each fuzz test.

Iteration count based fuzzing makes a given fuzz test finish successfully after that many different inputs have been tried. The number of iterations is per test, so adding more fuzz tests while keeping the same iteration count means it will take longer to finish successfully, since the new tests will be also given the iterations.

Open questions:

  • what about fuzz tests that might want more iterations or more time spent on them relative to each other?
  • should zig build default to --fuzz=1K or similar when fuzz tests are found? If this is set to a reasonably low value, it could be quite valuable in finding shallow bugs quickly. It also opens the door for reporting the seed rather than input and having developers reproduce the failure easily since the iteration count was low.
@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. fuzzing labels Feb 15, 2025
@andrewrk andrewrk added this to the 0.15.0 milestone Feb 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. fuzzing
Projects
None yet
Development

No branches or pull requests

1 participant