Finding a deep logic flaw used to cost weeks of skilled human attention. That cost has collapsed and attackers got the benefit at the same moment we did. This piece is about what we built in response.
The gap we had to close is the one that our scanners were never built for. They're great at what they do, but they only find what someone told them to look for: patterns, signatures, known-bad payloads. But some of the most critical flaws in a multi-tenant platform like ours are different. They're logic that is simply wrong: valid requests, correct syntax, an authorization decision made on the wrong basis. There's no pattern or signature to match against. Finding them means understanding what the system is supposed to permit, forming a theory about where it permits more than that, and testing it. That's expensive work, and the expense used to be a relatively effective deterrent for attackers.
It isn't expensive anymore, for us or for anyone else.
So in response, our team built a proof of concept for an agentic dynamic testing engine: autonomous AI agents that read our own source, hypothesize where controls like authorization and tenant isolation break, attack a running non-production instance, and then hand every claim to a deterministic layer that re-runs it end to end. That layer is the arbiter and the false-positive filter; it promotes only what it can independently reproduce and demotes everything else to a labeled lead, so a model's confident guess never reaches the report as a finding. It reaches the class of issue our existing tooling was never built to find.
We built it on a laptop.
There's no GPU farm behind this, no research team, and no infrastructure we had to stand up; it runs inside developer tooling the team already had. It fits inside our existing security budget, and the cost of a run is low enough that it doesn’t factor into deciding to do one.
Today, the POC is run by hand against core services and the validated findings it produces go to engineers for remediation. We plan to move the POC forward and wire it into our CI pipeline so it’s running on its own cadence.
None of what we built came from trying to spend less. It came from realizing that the model was never the bottleneck. Grounding and proof were, and those get built, not bought.
Here’s how we built it, and what running it taught us about where the models actually earn their place.
What we built
Almost none of the difficulty was in the code. It was in three decisions that came before it: what shape of system to build, what constrains it, and what runs inside that constraint. If you get the order wrong, you risk ending up with something powerful that nobody will act on.
Why we built a constrained ensemble
When we scoped this project, there were two credible architectures. Picking between them determined everything downstream:
- The maximalist option: Point a single very capable autonomous agent at the platform, give it broad latitude and unlimited compute, and let it learn your systems and roam. This takes minimal engineering on your side, and it improves every time the underlying model does. But it isn’t something you can plan to rely on as a control. It's general-purpose, so it spends much of its budget rediscovering what your own architecture documents already state. It's largely opaque, with findings arriving as assertions, leaving you to grade the agent's confidence instead of its evidence. And it's unconstrained, which means both a massive compute bill and a blast radius defined by whatever the agent decides to try.
- The constrained ensemble: Grounded in our real source. Discovery is distributed across more than a dozen models from six vendors rather than concentrated in one reasoner. Every claim terminates at a deterministic oracle that reproduces the exploit or discards it. It’s cheap enough to run on a regular cadence, small enough to audit line by line, and scoped by architecture, not by prompt.
The maximalist agent buys capability and pays in opacity. The constrained ensemble buys evidence and pays in engineering.
We built the constrained ensemble with the bet that a smarter model doesn’t make an unproven finding actionable, and no amount of reasoning substitutes for reading the authorization path. We decided to spend our effort on the scaffolding, not on procuring the single best brain.
The boundary comes first
The piece that came first isn’t the most exciting, but it’s the most important. If you take the capability argument in this piece and skip this section, you haven't built a security testing platform; you've built an unsupervised attack system and pointed it at your own company.
Every path to a target in our engine runs through a single policy-gated execution core that’s rate-limited, with every write scoped to a disposable canary object which the tool itself created, then restricted to non-production environments. Production is strictly read-only; writes and autonomous agents are restricted at the gate. It’s not just a rule in a runbook;it’s the only code path that exists. The scope lives in the architecture, not in the prompt, because a prompt is a request and an architecture is a constraint.
Autonomous agents with real exploitation capability are only as safe as the boundary around them. Build the boundary first. If you can't state your enforced blast radius before you start, you aren't ready to start.
Inside the boundary: the pipeline
Your initial instinct may be to take one strong model and scale it out to multiple parallel agents. Resist it. Blind spots correlate inside a model family, so multiple agents with identical instincts can walk past the same flaw and report high confidence doing it. Coverage looks like a compute problem, but it’s actually a diversity problem. We run more than a dozen models across six vendors, which turns one model's opinion of your surface into a matrix. Diversity buys precision and coverage: a hypothesis raised independently by models from two different vendors gets corroborated and sorted first. The model pool is what the reasoning stages draw on. The recon and verification stages are deliberately model-free. The pipeline itself is six stages, and the five that touch the target reach it only through the boundary:
- Deterministic recon runs before any model does. We map the full callable surface, documented or not, and fire the cheap probes at all of it. This is mechanical work that needs no reasoning. Agents get those results as their starting input. It's the same as handing a pentester the recon instead of asking them to start cold.
- Scouting agents read the source, not just the API. White-box agentic testing beats black-box, and it isn't close. When a scout can read the authorization path and the middleware around it, its output stops being a guess about behavior and becomes a claim about implementation. You already trust these models with your code in your IDE; trusting them in your test harness isn’t a new decision.
- Exploit agents prove the hypotheses. When a model proposes a claim, its exploit isn’t taken as proof. A deterministic oracle that shares none of the model's incentives has to reproduce it end to end, and a separate skeptical model, never the proposer, tries to refute it. A model grading its own hypothesis grades generously, so we don't let a model be the judge at all.
- Verification is deterministic and can throw the AI's work away. A finding is only proven when a non-AI oracle independently reproduces the exploit end to end, and that same oracle re-runs each confirmed proof several times and drops anything that passed only once — a lucky or cached response never ships. This is also where false positives die: a claim the oracle can't reproduce is demoted to a labeled lead, duplicate reports of one bug collapse into a single finding, and a claimed information leak only counts if a normal, benign request doesn't produce the same error. Otherwise the agent has found how the application always behaves, not a flaw. Everything that fails to reproduce is queued for a new oracle or a human red-teamer. Demoted, never silently dropped.
- A reproduced exploit still isn’t automatically a bug. Sometimes the behavior is intended, and only someone who knows the system would catch it. So before a proven finding ships, a panel of separate models re-checks it against three things: our own design docs, the source (including the upstream services the API gateway delegates to, where the real authorization often lives), and a fresh live probe. A finding is set aside as by-design only when all of them agree, with evidence, and none defends it as a real break; a split or uncertain verdict leaves it in the report. The pattern this catches: a tenant can mint an invitation that looks like a cross-tenant write, right up until you notice that the actual control is that only the receiving tenant’s admin can accept it. Proof gets you past, “is it reproducible?” This gets you past, “Is it actually wrong?”
- Reporting ships remediation, not detection. Every proven finding carries the affected operation, source path, severity, reliability, weakness mapping, an exploitation scenario, a proposed fix with blast radius, and the replayable proof of concept. No re-scan, no re-triage, no argument about whether it was ever really exploitable.
What we learned
With the proof of concept working, we could now measure which models were earning their place. What we learned changed which models we run and what we ask each of them to do.
The best model overall wasn't the best model for this
In one run, our top-producing model surfaced roughly seven times the validated leads of the frontier model holding the primary reasoner role. It wasn’t the most expensive model in the pool, or the one that wins coding benchmarks. It was an open-weight model.
The spread was also enormous and unpredictable from the outside: the top contributor produced roughly fifty times the validated findings of the weakest and one model, blocked by its provider's filter, produced none at all. Rank order had almost nothing to do with published benchmark position or price per token. If you're selecting the model behind your security tooling on general capability score, you're selecting on a proxy that doesn't hold here. The only way to know is to measure on your own surface.
Where guardrails trip, an open-weight model picks up the work
Several frontier providers apply real-time filters that decline offensive security work. In our runs, that included authorized, owner-initiated testing against our own non-production systems. One model in the pool returned zero findings for that reason — not because it lacked the capability, but because it wouldn't accept the task.
Those filters are a reasonable product decision. A hosted provider can't verify that the person asking it to find an authorization bypass owns the system in question, and the consequence of getting this judgement wrong is severe. The filters aren't the problem.
The problem is how the refusals surface. A declined unit of work doesn't throw an error. Nothing crashes, the run completes, and unless you're watching for it, the only trace is an absence. A result that produced nothing looks identical to a result that found nothing.
So we don't let the hypothesis die at the model refusal. Every declined unit is detected, logged, and re-dispatched to an open-weight model running under terms that permit authorized security testing. The work changes hands rather than disappearing, and the refusal itself gets surfaced in run health alongside verification downgrades and blockers, so we can see the rate and trend.
In our runs, the open-weight backstop did more than we expected. In a single run, 64 exploit attempts were refused on policy grounds. Together with timeouts and other errors, 89 were recovered by the backstop chain; the same mechanism catches work that fails for any reason, not just refusals. Without somewhere for that work to go, those hypotheses are gone, but the report would still come back looking complete.
None of this is a way around any of the models’ safety controls. It's the position you occupy with every offensive security tool you own: you hold the authorization, you define the scope, you carry the accountability, and the architecture enforces all three. The difference is that a third party's evolving policy stops being a silent input to your own assurance. If you take one operational recommendation from this piece, it’s this: count the refusals and have somewhere for that work to go, otherwise you can't claim the coverage.
What we changed: frontier for depth, open-weight for breadth
Model benchmark ranking didn't predict results, and no single model would have given us the breadth and depth of coverage we gained through our multi-model approach. The two tiers of models earn their place in different roles.
Frontier models earn their place on depth. When an agent holds one specific theory and needs to chain three conditions to prove it, they're noticeably stronger, and better at reading unfamiliar code and reasoning about an unfamiliar weakness. We don’t hard-assign them to a stage; every model scouts every risk lens. But the hardest multi-step chains are where a frontier model most often makes the difference.
Open-weight models cover the breadth. They produced the bulk of our validated leads at a fraction of the cost, which matters when your scout stage is proposing thousands of hypotheses. And they didn’t stop at breadth; the single highest-yield model at the exploit stage was open-weight too. They also come under terms that don't prohibit authorized security testing, so the work simply runs. Nothing decides mid-run whether your authorized test of your own system looks acceptable.
What to take away
None of this needed a GPU cluster, a research team, or a seat in a frontier lab's restricted-access program. It needed an enforced execution boundary, deterministic recon feeding a reasoning layer, your own source in the loop, models from more than one vendor, and an oracle with the authority to throw the models' work away. It's less about having access to the best model and more about running enough of them that no single vendor's blind spots or policies become your own.
If you're building something like this, start with one weakness class that your team already understands and wire one oracle for it. Then prove a single bug end to end: hypothesis, exploit, independent reproduction, a ticket an engineer closes and can verify. Once you prove it holds for a single weakness class, you can add lenses and models to a machine that already knows how to tell proof from suspicion. If you start with the model pool, you risk building something impressive that nobody trusts.
From there, the work’s focus shifts from capability to cadence. A tool someone has to remember to run is still a snapshot in time; you want a version that's wired into the CI pipeline, running on whatever cadence the economics support so that coverage is a property of the pipeline and not a decision someone has to make.
That’s where our work goes from here: the engine is already providing value today, but wiring it into the CI pipeline is what turns a capability into a control.
Frontier capability turned out to be the easy part to source. What's hard is everything that decides which claims survive. If you build that first, the models become interchangeable, which is exactly what you want them to be.
