Open-sourcing pentimento: an AI audit pipeline that has to prove what it finds

Today we're open-sourcing pentimento, an AI-orchestrated audit pipeline for Solidity. It's MIT-licensed and available now: github.com/danogurtsov/pentimento. We built it a few months ago, used it on our own work since, and are only now getting around to publishing it properly.

The short version: it doesn't just ask a model to find bugs. It resolves what's actually deployed before it looks for anything, routes cheap deterministic checks ahead of every expensive LLM call, runs a structured adversarial protocol that computes its own verdict in code rather than trusting the model's self-report, and — for anything it confirms — writes a real test and executes it. If the proof-of-concept doesn't reproduce, the finding doesn't ship as confirmed. That last part turned out to be the difference that mattered most.

Where this started

Before writing a line of pentimento, we did the boring part first: we counted what already exists. 84 tools, 16 benchmarks, 60+ academic papers later, a pattern was hard to miss. Independent recall sits at 30-40%. The best independently verified precision for a production AI audit tool is still barely above half. And a large share of tools that publish an impressive number turn out to be grading themselves: the same model that found a "bug" is the one deciding whether it's real, with no independent check and no executed proof standing behind the claim. A separate pass we ran across five contests and 145 tool runs found something just as inconvenient: one run of the same tool catches barely half of what it's capable of finding — LLM output is noisy enough that a single pass is a coin flip on what you get.

None of that is a reason to give up on the idea. It's a reason to be specific about which part of "AI audit tool" is actually doing the work, and to stop taking a model's word for the part that matters most: whether a finding is real. We paired that landscape review with a second one — how organizations that actually run AI agents in production structure the trust layer around them (approval gates, cost governance, independent verification, evals with error bars, not vibes). A real audit report doesn't say "probably vulnerable" — it says confirmed, or it says unresolved and hands it to a human, and it never quietly drops a rejected claim into the same pile as a real one. That's the report shape pentimento tries to earn, not just format.

Repo files aren't deployed contracts

The first thing pentimento does has nothing to do with LLMs. A .sol file boundary almost never matches a deployed contract's real boundary — a proxy and its implementation are two files and one contract; a diamond can be a dozen facet files and one logical unit; a factory's template contract isn't the same thing as the instances it stamps out a thousand times over. Ask a model to reason about "the contract in this file" and you've already lost information the file system never had in the first place.

So before anything else runs, pentimento resolves a Canonical Deploy View: one unit per deployed contract, with every merge made explicit. It classifies each contract's node type from its ABI and resolves six structural primitives — proxy/implementation merges, diamond/facet merges (matched by directory proximity, since a real monorepo reuses facet names across unrelated diamonds more often than you'd hope), dispatcher-proxy merges, getter- and event-enumerable factory resolution, vault/share-accounting shape, and singleton/logical-entity detection including counter-based id schemes, while correctly leaving genuine clone factories alone. It isn't the first time we've built on "resolve the real structure before reasoning about it, and only trust what you can verify" — a similar principle, applied to live on-chain state instead of a repo, sits underneath an earlier project of ours. This one actually came first.

None of this was tuned against toy fixtures alone. Running it against real, independently chosen repositories — historical incident corpora, multi-facet diamonds that reuse facet names across directories, clone-factory account systems, a real derivatives protocol with real library dependencies — found and fixed real misclassifications: multi-contract files silently attributed to the wrong contract, cross-file name collisions where nine of ten files in a directory reused a generic test-contract name, bare interfaces and libraries wrongly promoted to their own deployable unit, and ERC-6909 multi-token contracts that overlap ERC-20 by function name but never by exact signature. A second converter path resolves the identical CDV output directly from a live chain instead of a repo — same output shape, so nothing downstream needs to know or care which one produced a given unit.

Cheap signals before expensive reasoning

Every detection call is preceded by deterministic, zero-cost analysis that decides what to even ask a model, and how hard to look. An Engine Selection Matrix unions a unit's node type and proxy kind independently to pick which threat engines apply and at what depth — an upgradeable vault gets full coverage on every relevant axis instead of being squeezed into one label. A guard consistency check looks at every state variable with three or more writer functions and flags the minority that don't share a common guard, with no LLM involved at all — the anomaly gets pre-flagged as a candidate for the model to look at, and it's written out standalone so the signal exists even without ever making a call. A state synchronization check clusters state variables that are consistently co-modified and flags any function that touches only one side of an established pair.

Two routing layers sit on top. A cheap regex pass suggests domain-specific checklists (lending, AMM/DEX, yield-vault) from genuine functional co-occurrence — deposit next to borrow means something, deposit alone means nothing — but the actual activate-or-skip decision is a dedicated, deliberately cheap LLM call that sees only function signatures, never bodies, and every domain gets an explicit recorded outcome either way. On a real production vault router, this caught something worth noting: the router correctly activated the right checklist on files where the cheap pre-scan found nothing at all, reasoning from signatures alone — and just as correctly skipped domains that plainly didn't apply, with real justification instead of a rubber-stamped label. The second routing layer is model-aware: a structural complexity signal (imports, function count, line count) flags dense contracts for automatic escalation to a stronger model, because a cheap model quietly running out of attention on a large real contract is a failure mode that doesn't announce itself.

A verdict is computed, not asserted

This is the part we spent the most time on, because it's the part every "AI finds bugs" pitch conveniently skips over: how do you know the finding is real? A surviving candidate goes through a structured, adversarial false-positive-check protocol — data flow analysis, feasibility verification (bounds claims have to walk an explicit constraints-to-proof chain, not a paragraph that merely sounds right), impact assessment, a proof-of-concept sketch, a fixed set of devil's-advocate questions designed to argue the finding away, and six mandatory gates.

The model only ever reports per-gate pass or fail, with a reason. TRUE_POSITIVE requires every single gate to pass — the verdict itself is computed by code, never asserted by the LLM, and a gate the model's response never addresses is recorded as an explicit failure rather than silently assumed to pass. Run against a real production allocator contract, this is exactly what caught the distinctions a shallower re-read misses: correctly rejecting a reentrancy-shaped finding because the primary control (a max-withdrawal check) was intact and the flagged path was only defense-in-depth; correctly rejecting a low-level-call finding because checks-effects-interactions was already applied; and correctly confirming the one finding that actually held up across every gate. Findings whose shape calls for it — ambiguous claims, cross-contract paths, race conditions — route to a full multi-phase verification pipeline instead of the single linear pass, with automatic escalation to a stronger model when a cheaper one fails to produce a properly formatted verdict.

On top of that: an optional second, independent verifier, where confirmation requires both to agree and a single dissent flips the verdict. And a five-tier evidence ladder replacing a bare model self-report with something that actually weighs evidence — a reproduced, executed proof-of-concept outranks everything else; a self-report alone can nudge the number, never dominate it.

Code executes the proof, the model doesn't get the last word

For a confirmed finding, pentimento asks the model for a complete, compilable test — never pseudocode, never a plausible-sounding paragraph — grounded against a real, already-working test file from the target project's own suite when one exists, so the model inherits a working setup instead of inventing constructor wiring from nothing. The generated test is written to disk, run for real, and deleted regardless of outcome. The verdict is the test runner's own exit code, not the model's word for it.

Here's what that looked like end to end, on a real audited protocol: the scout stage raised four candidate findings on a live allocator contract. The adversarial verification pass confirmed one — an "empty supply queue blocks all deposits" denial-of-service — and rejected the other three with specific, sound reasoning: one was defense-in-depth sitting behind an intact primary control, one was already protected by checks-effects-interactions, and one was a revert-on-bad-input case with no real security impact at all. For the confirmed finding, the pipeline generated a Foundry test reusing the project's own integration-test harness, ran it for real, and it passed — [PASS] testAllocatorCanBlockAllDepositsWithEmptySupplyQueue(), exit code 0. Not a model saying it should work. A test runner saying it did.

Where we landed

Building this pushed us toward a couple of specific opinions about how these pipelines should be put together, not just how pentimento itself works. We think a structural fix to what a model is even asked tends to buy more reliability than stacking another verification pass on top of it — it's cheap, it's always on, and it compounds with everything downstream, where an extra voting or ensemble step only helps the one place it's applied. We also think an automated judge grading a pipeline's own output should come from a different model than the one doing the work, full stop — a judge too close to the detector tends to agree with it more than it should, in ways that don't show up unless you go looking for them. Both opinions are baked into pentimento's own defaults now, not just written down somewhere.

There's more we'd like to add than we've had time for — broader statistical confidence behind the detection numbers, a genuinely independent model provider wired in as the default judge rather than an opt-in, and a larger private, contamination-free fixture set to keep testing against. None of it felt urgent enough to hold the release for, and it's exactly the kind of thing an open-source release tends to speed up.

The parts that don't touch the finding itself, but decide whether you'd trust it

A few things exist purely so the pipeline can be run against real, adversarial, untrusted code without becoming an attack surface itself. A report separates confirmed findings from disclaimed, genuinely-unresolved leads from rejected claims — rejected claims get dropped entirely rather than shown anywhere as noise. Nothing final gets written without an explicit human approval step, and approval records a hash of the exact report text shown at that moment, so a later mismatch is detectable rather than merely claimed. A shared cost ceiling actually enforces — once tracked spend crosses the line, the next call is refused before it happens, not after. A deterministic pre-scan checks every piece of source for injection attempts before it ever reaches a prompt — both blunt "ignore previous instructions" attempts and the subtler move of mimicking the tool's own prompt template to make injected text look like trusted output. And because the proof-of-concept step shells out to a target project's own test suite — genuinely untrusted, possibly adversarial code — every LLM provider secret is stripped from that subprocess's environment first, and the whole step hard-refuses to run at all against a project with Foundry's ffi cheatcode enabled, since that cheatcode is arbitrary shell execution disguised as a test run. We found that gap by checking, not by assuming it away — one of our own real test fixtures actually has ffi enabled, which is exactly the point of checking.

Try it

pentimento is free, open-source, and MIT-licensed:

git clone https://github.com/danogurtsov/pentimento
cd pentimento
uv pip install -e ".[dev]"

pentimento breadth-pass <path-to-contracts> --llm claude-cli:haiku --out out/breadth --solc <path-to-solc>

It's model-agnostic — your own Claude Code subscription with no API key at all, the Anthropic API directly, or any OpenAI-compatible provider plugs in behind the same interface. The scope we're comfortable claiming today, and nothing more, is written up plainly in the repository itself.

If you run it against something real, we'd genuinely like to know what it finds — and, just as usefully, what it doesn't.