Tools for Cybersecurity

Practical guide · November 17, 2026

AI vulnerability discovery versus fuzzing: compare the whole search process

Evaluate AI discovery and fuzzing by the additional validated findings each contributes under a defined budget. Do not compare an agent given a suspicious change with a fuzzer asked to explore an entire program, then call the result a universal contest between AI and traditional testing. The target, available information and failure detector are part of the experiment.

For teams already running fuzzers, the useful question is where an agent changes the search: identifying a neglected entry point, proposing a security invariant, constructing a harness, investigating a crash or finding a defect that the existing campaign misses. These contributions need different evidence and should not share one undifferentiated success counter.

Start with the mechanics, not the labels

Fuzzilli's project documentation describes a coverage-guided fuzzer for dynamic-language interpreters. It mutates an intermediate representation, translates programs to JavaScript, executes them, evaluates samples and minimizes interesting or crashing programs. It is not simply sending random bytes at a service.

That architecture gives you several separate things to inspect: the generator's vocabulary, the corpus of retained inputs, the instrumented target, and the evaluator that decides what deserves attention. In a local comparison, record all of them rather than identifying the baseline only as “fuzzing.”

For an agent, record the analogous configuration: code and documents it receives, model and revision when available, tools it can call, feedback it receives, allowed execution environment and stopping conditions. If it may edit harnesses or consult external information, record that too. These are our proposed evaluation controls, not a description of every agentic research system.

What the BigSleep comparison actually establishes

Google Project Zero's October 27, 2026 comparison examines Fuzzilli and BigSleep on V8. Both found actionable vulnerabilities. The reported production overlap was lower than the overlap found in a separate historical-rediscovery experiment. The author concludes that the methods are complementary and says a meaningful cost comparison remains difficult.

Those observations support retaining both approaches in that setting; they do not establish a winner for SaaS authorization testing or another language runtime. The historical experiment also restricts bugs by build and runtime conditions. Its denominator is not every vulnerability the fuzzer ever found.

Our practical inference is to evaluate incremental contribution on your own target rather than importing a headline result. The study's distinction between production discovery and controlled rediscovery is particularly useful: what a system could find under a special experiment and what it actually finds during normal operation are different questions.

A harness defines the reachable experiment

Google's OSS-Fuzz target-generation work places an LLM in a build-and-feedback loop for fuzz targets. That is a concrete example of combining methods: AI proposes a way into the software, while the fuzzing machinery explores inputs through that interface.

For a pilot, inspect the wrapper before celebrating its crashes. Does it initialize required state, respect the API's ownership rules and reach the intended parser or subsystem? Does it call an internal function in a state that a real caller could create? Keep the generated wrapper and any repair attempts so a reviewer can answer those questions.

Consider a hypothetical image decoder that expects a session object created by its initialization routine. An agent-generated harness passes an uninitialized object directly into a later operation and obtains a crash. That is not yet evidence of a defect in the supported decoding path. Revise the harness to obey the contract, preserve the same relevant input, and determine whether the failure remains reachable. The example is illustrative, not a reported vulnerability.

Do not go too far in the opposite direction: an internal entry point may still represent a real security boundary. The reviewer must establish its callers and permitted state, not dismiss it merely because it is internal. Document the reasoning either way.

In your evaluation plan, define the oracle: the observation that marks an input as a candidate failure. It might be a memory diagnostic, a crash, an assertion about tenant isolation, or disagreement with a reference implementation. State which one you used and why it relates to the security question.

A proposed authorization invariant, for example, can be stated as “an identity outside the resource's tenant must not receive its protected content.” Merely seeing an HTTP error or a new code path does not evaluate that invariant. A passing result must reflect the protected operation, not the text of the error message alone.

Require reviewers to distinguish more reachable behavior from more security defects. Additional coverage is useful evidence about the search but should not be added to the vulnerability count. Likewise, a crashing input remains a candidate until its relevance and distinct root cause have been examined. Apply the same acceptance rules to agent and fuzzer outputs.

Keep discovery, reproduction and repair separate

CyberGym provides real-world vulnerability-analysis tasks with executable validation. Its task artifacts and evaluation conditions matter. A successful reproducer on a known-vulnerability task does not by itself measure unrestricted discovery in an unfamiliar repository.

For an internal experiment, define three tracks. In discovery, provide the application and agreed architectural context but withhold the answer. In reproduction, provide the candidate claim and ask for evidence. In repair, provide an accepted finding and assess the patch. Do not mix those tracks in one reported “find rate.”

If you seed a historical test with the introducing commit, record that information advantage. If you give a fuzzer a newly improved harness based on an agent's finding, record the hybrid workflow rather than crediting the baseline with an independent discovery. These controls make results interpretable; they do not eliminate all possible leakage from public historical examples.

Run a small incremental-yield experiment

Our suggested experiment begins with a reproducible baseline, not a deliberately weak comparator. Freeze the target revision, build, starting corpus, existing harnesses, enabled checks and authorized execution policy. Choose a bounded component and a review budget that the team can actually support.

Run the existing fuzzing setup, then evaluate agent-assisted work against the same target. If resources permit, separate direct agent investigation from agent-generated harness improvements. Otherwise report the combined intervention honestly. Record any differences in target access or information rather than pretending the runs are identical.

Use a shared adjudication ledger with these fields:

Keep repeated trials separate until you define how to combine them. One run with no findings cannot establish that a method is unable to find a class of defects. Conversely, repeatedly querying a system until one run succeeds should not be represented as a single-attempt result.

For economic interpretation, propose an incremental metric: additional confirmed root causes beyond the baseline, together with the additional compute and reviewer effort. Report cost components separately when your accounting cannot reliably combine them. Do not assign a universal monetary value to every bug or infer production risk reduction from counts alone.

Interpret a negative result as a diagnostic

If an agent produces no validated finding, inspect where the process stopped. Did it build the target? Reach the intended component? Generate test inputs? Obtain useful feedback? Produce a candidate that reviewers rejected? Each answer suggests a different next experiment.

Apply the same discipline to fuzzing. A run that never enters the relevant state is different from a run that exercises the state without triggering the chosen oracle. Neither outcome proves the component secure. Preserve the configuration and observed limitation so later work can build on it rather than repeat it.

The strongest use of AI may be to improve a durable testing asset: a reviewed harness, a regression test or a clearer security invariant. Keep those artifacts in the ordinary engineering workflow after the pilot ends. Continue using the baseline while evaluating whether the new method contributes enough additional evidence to justify its cost.

For the broader workflow choices, see AI vulnerability discovery tools. For the acceptance decision on an individual result, use how to validate AI security findings.

Evidence reviewed November 17, 2026. AI-assisted synthesis of primary research and project documentation; no fuzzers, agents or benchmarks were run for this article. Experimental design recommendations and the decoder scenario are original editorial illustrations, not measured performance results.