Benchmarks
How we measured DMN
We had Claude Code agents track down real bugs, with and without DMN, and wrote the rules down before any run. This page has the numbers, how we got them, where DMN didn't help, and what the numbers don't show.
Finding the code behind a bug, on code the model could rarely place without tools, and finding it about as often as without DMN. Public code is one campaign of 56 bug reports, and its token saving was within noise. Our own code's ranges run from the lowest to the highest of three campaigns on the same 14. On a popular project, where the model named the spot with no tools for 15 of 36 tasks, DMN saved a few requests and no tokens.
Results
Every result here measures the same job: Claude Code gets a real bug report and must name the function that needs the fix, with DMN and without it, under rules written down before the first run. The newest come first, and each says when it ran, on what code, with which model and setup.
On public code
2026-10-02 · Claude Code on Claude Opus 5.5 · zeroclaw and vortex, public Rust code newer than the model · 56 bug reports, each run twice with and twice without DMN · DMN's skill, as it ships
To check our own-code results on code we had never tuned DMN on, we took 56 new bug reports from two public Rust code bases newer than the model's training data (zeroclaw, 51; vortex, 5). With DMN agents made 11% fewer requests (95% interval 3–18%) and found the right code as often (107 against 108 of 112). The token saving, 8%, was within noise (an interval from 18% fewer to 4% more). A follow-up that changed two of DMN's defaults saved nothing either, and we rolled those changes back. So we claim a token saving only on our own code.
On DMN's own code
2026-09-30 · Claude Code on Claude Opus 5.5 · DMN's own code, private · three campaigns on the same 14 bug reports, each run twice with and twice without DMN · DMN's skill: revision 2 in campaign 01, revision 5 (as it ships) in 02 and 03
| Campaign | Requests (paired) | 95% interval | Tokens (paired) | 95% interval | Found it (without / with) |
|---|---|---|---|---|---|
| 01 | −15% | 4–25% fewer | −19% | 6–30% fewer | 27 / 25 of 27 |
| 02 | −16% | 6–25% fewer | −11% | 24% fewer to 5% more | 27 / 27 of 28 |
| 03 | −26% | 18–33% fewer | −25% | 15–34% fewer | 26 / 26 of 28 |
The headline ranges at the top of the page are the lowest and highest of these three: 15–26% fewer requests, clear in every campaign, and 11–25% fewer tokens, clear in two (campaign 02's interval includes no saving). Each figure compares every task with itself, with and without DMN, averaged over tasks (a geometric mean). "Found it" counts runs that named the right function among their top three answers. The 14 tasks are the September benchmark's bug reports (below) that Opus 5.5 couldn't place with no tools.
DMN's own code is also the code DMN's search was developed on. The public-code campaign above checks the result on code it wasn't, and saved less.
Earlier: the September benchmark, on three code bases
2026-09-20 and 21 · Claude Code's search agent on Claude Opus 5, told to use DMN (before the skill shipped) · DMN's own code (31 bug reports), vortex (30) and ruff (36) · each run twice with and twice without DMN
Each bar is what an agent spent with DMN, as a share of what it spent without. Shorter is better; the line is "the same as without".
| Code base | Tasks | Found it (without / with) | Median requests | Median tokens | Requests (paired) | Tokens (paired) |
|---|---|---|---|---|---|---|
| DMN's own code | 31 | 59 / 58 of 62 | 7.5 → 5 | 267k → 189k | −25% | −22% |
| vortex | 30 | 58 / 58 of 60 | 8 → 5.5 | 301k → 210k | −21% | −16% |
| ruff | 36 | 69 / 67 of 72 | 8 → 6 | 251k → 209k | −8% | ±0% |
"Found it" counts runs that named the right function among their top three answers. The paired columns compare each task with itself, with and without DMN, averaged over tasks (a geometric mean), so they differ from the change in medians. The rule written before the runs let DMN's success rate fall by up to 10 points; it fell by at most 3. Runs on 2026-09-20 and 2026-09-21.
Whole tasks, on someone else's benchmark
The results above measure one step: finding the code. To measure whole tasks we ran DMN on TokenBench, an independent benchmark (MIT licensed) of 55 coding tasks graded by hidden tests the agent never sees. Each task is fixed or built end to end, and the cost is Claude Code's own dollar figure. We wrote our decision rules before the runs, paired every task with itself with and without DMN, and report intervals. Two campaigns ran on 2026-10-03, about 1,000 runs.
| Model | Setup | Cost per task vs. without DMN | Quality (hidden tests) |
|---|---|---|---|
| Claude Sonnet 4.6 | DMN as shipped | −15% (95% interval 8–22%); −18% in the first campaign | +2.2 points |
| Claude Sonnet 4.6 | DMN, leaner briefing (in review, not shipped) | −29% (interval 19–39%) | +3.8 points |
| Claude Opus 5.5 | DMN as shipped | +5% (interval +2% to +10%) | −0.7 points |
Per-task geometric means of the dollar cost. The leaner briefing's interval is 98.3% (three comparisons were
registered); the others are 95%. Quality is TokenBench's own score (85% hidden tests, 10% visible tests, 5%
integrity), out of 100. Full results, pre-registrations and harness:
tools/bench/tokenbench-2026-10/ in our repository.
Where the saving comes from: Sonnet made 29 file reads a task on its own and 14 with DMN's lean briefing, and finished faster (12% less wall time with DMN as shipped, 26% with the lean briefing). On the same day, caveman, an output-compression plugin, saved about as much as the lean briefing (−28%) at slightly lower quality. Opus 5.5 already makes under 2 file reads a task on this benchmark, so there is nothing to save, and DMN's guidance costs it about 5%.
What these tasks don't test is search: they are small repositories, and agents used DMN's search in 8 of about 500 DMN runs. A bank of whole tasks on a large repository the models have never seen is built and validated; that is the next campaign.
What a run looks like
In the September benchmark on DMN's own code, the median run without DMN took 7.5 requests and 11 tool calls: search, read a file, search again, read another. With DMN it took 5 requests and 6 tool calls, and asked DMN once.
DMN doesn't hand over the answer. Its results point the agent at the right part of the code and the right words for it; the agent still reads the code and decides. Tasks where DMN's first results missed the right file gained the most on this code base, because being pointed at the right area was enough.
Where the gain is
Splitting the September benchmark's tasks shows where DMN helps and where it doesn't.
| Tasks | Count | Requests | Tokens |
|---|---|---|---|
| DMN's code: no word in the report points grep at a few files | 23 | −29% | −28% |
| DMN's code: a word in the report matches five files or fewer | 8 | −10% | ±0% |
| DMN's code: Rust and other | 21 | −29% | −29% |
| DMN's code: the TypeScript GUI | 10 | −14% | −4% |
| vortex: the fix touches several places | 10 | −32% | −30% |
| vortex: the fix touches one place | 20 | −16% | −8% |
The pattern: DMN helps most when a bug report gives grep nothing to hold on to, and when the fix spans several places. When one word in the report already names the file, grep is as good.
The newer campaigns agree. Campaign 03 also ran 10 vortex bug reports, and across all 24 tasks the harder ones, sorted before the run, took about 25% fewer tokens with DMN (interval 7–39% fewer); on the quicker ones the difference was within noise.
The misses
Code the model can already place
For the September benchmark we also gave the model each bug report with no tools at all and asked where the bug was. It could already name the spot for 3 of 31 tasks on DMN's code, 2 of 30 on vortex, and 15 of 36 on ruff, a popular project it has most likely seen in training. On ruff it often barely needs to search, so DMN had little to save. The newer campaigns leave such tasks out before they start.
Results full of test snapshots
On ruff, 44% of what DMN returned was test files, most of them snapshots that quote the same error text as the bug reports. That pushed the real source down the list. On DMN's own code, test files were 1%.
A bar we set, and missed
Before the September benchmark we said a headline result needed both ratios at 0.75 or lower. DMN's own code came in at 0.754 and 0.784: clear and consistent, but just short. So we call it modest.
Speed
The counts above are about work. These are about time, each measured on its own, with where it came from.
| What | Result | How it was measured |
|---|---|---|
| A search, answered | 5–12 ms | Median over 17 developer questions each on Django (10 ms), Vite (5 ms) and ruff (12 ms), full-project indexes, on a desktop with a Ryzen 9 9950X3D and an RTX 4070 Ti SUPER. September 2026. |
| A search, CPU only | 20 ms | Median over 30 first-time questions and names on a 44-file sample project, on the CPU alone with the EmbeddingGemma model (26 ms at p90). Asking the same thing again takes about 1 ms. |
| From saving a file to finding it | 2.2 s | Median over 24 saves of a new function into the same sample, CPU only, found by its name (2.2 s at p90); 2.3 s over 12 more saves found by meaning alone. 2 s of it is a set pause that gathers a burst of saves into one update (DMN_CONTINUOUS_DEBOUNCE_MS); the update itself took about 0.15 s. September 2026. |
| One agent round trip | ~3 s + reply | Across 62 recorded coding runs, each request to the model took about 3.1 seconds plus 30 ms per token of reply. |
Put together: finding the code took 11% fewer round trips on public code and 15–26% fewer on ours, and each one skipped is a few seconds the agent isn't waiting. On whole tasks (TokenBench, above), Sonnet 4.6 finished with 12% less wall time with DMN as shipped and 26% less with the lean briefing.
The race, run for real
The races on the home and engine pages replay real runs. Claude Code (Sonnet, headless) was asked "where do we
rate-limit logins?" on a 44-file sample project: three times without DMN, three times with DMN as it ships
(terse answers on), and three times with terse answers off, in turn. Every arm could grep, glob and read; the DMN
arms could also run dmn search. The protocol was written down and committed before the first run,
and all nine runs are reported.
| Median of 3 runs | Correct | Requests | Tokens | Seconds |
|---|---|---|---|---|
| Without DMN | 2/3 | 3 | 16,462 | 6.2 |
| With DMN | 3/3 | 2 | 17,317 | 3.9 |
| With DMN, terse answers off | 3/3 | 2 | 15,724 | 4.3 |
Tokens count cached input in full, as above. The terse-answer rule adds about 800 tokens to every request; on a two-request lookup that costs more than the shorter answers save, while on real bug reports it cut tokens by 23% (see what each part adds). One run without DMN named the right file and function but wrote the path with backslashes, which the check written down beforehand counts as wrong. One small project, so the benchmark above is the fuller measure.
What each part adds
DMN has a few parts that could save tokens: its search, a rule that makes the agent answer tersely, a budget on how much code a search returns, and serving it over MCP instead of as a skill. We measured each one on and off: real bug reports on DMN's own code (14 tasks, each twice), Claude Code on Opus 5.5, 196 runs, protocol written down first. Every setup found the right code in 26 of 28 runs.
| Setup | Tokens | 95% CI | Against | What it means |
|---|---|---|---|---|
| DMN's search, as a skill | 0.85× | 0.71–1.01 | no DMN | 15% fewer tokens, not clear on its own here; 16% fewer requests (0.74–0.95), which is clear |
| + terse answers (the compact rule, now on by default) | 0.77× | 0.61–0.97 | the skill alone | 23% fewer; 0.65× against no DMN (0.53–0.80), which is DMN as shipped now |
| Search budget of 5,000 characters | 0.92× | 0.80–1.06 | the skill alone | no clear effect: agents raised the budget themselves |
| Search budget of 3,000 characters | 0.99× | 0.86–1.14 | the skill alone | no clear effect, for the same reason |
| DMN over MCP instead of the skill | 1.12× | 0.97–1.29 | the skill alone | no clear effect: the agents never called the MCP tools |
Terse answers are on by default for Claude Code; [skill] compact = "off" turns them off. A blinded check rated them about as
useful as the full ones: 4.5 against 4.6 out of 5, a difference inside the margin set beforehand. They sometimes
leave out a secondary detail, such as a suggested fix.
One comparison is missing: a plain agent told to answer tersely, without DMN. So the 0.65× is a result for DMN as it ships, search and terse answers together. It doesn't say how much of that saving the terse rule would give an agent on its own.
How we ran it
- The task: the agent gets a real bug report and must name the function that needs the fix. Every task is a real bug, fixed in a real commit after the code snapshot.
- The agent: Claude Code with read-only tools, a fresh agent for every run. On Claude Opus 5.5 (effort xhigh) for the public-code and own-code campaigns; Claude Code's search agent on Claude Opus 5 for the September benchmark.
- Without DMN: grep, glob and file reads. With DMN: the same, plus DMN's search, which the agent asks in plain words. The newer campaigns used DMN's skill, as you'd install it; the September benchmark, before the skill shipped, told the agent to use DMN.
- Fair tasks: two reviewers agreed the right answer, and the bug reports were written without looking at the fix, so they don't name it.
- Code the model can't place: each bug report went to the model with no tools first. The newer campaigns left out every task it could place; the September benchmark reported them (see the misses).
- Every task, twice in each arm. Nothing was dropped for being too easy or too hard.
- Counted from each transcript: requests to the model, and tokens (input, cached input and output).
- Registered first: the tasks, the pass rule and the bar for a headline result were written down before any run, and not changed after.
Checks
- Public code: the October campaign repeated the own-code result on code DMN was never tuned on. Requests held up; tokens didn't.
- No tools at all (September): tasks the model could answer with no tools were set aside in a second pass; the result held without them.
- Home advantage (September): on tasks whose files had ever been used to tune DMN, the gain was about zero. All of it came from files never used for tuning.
- One task carrying it: dropping any single task left the result about the same.
- Noise: two runs of the same task often differ by a request or more, so we report averages over tasks, never single tasks.
What it doesn't show
- It measured finding the code, not a whole coding session. Sessions also write and test code, where DMN doesn't change much. On one set of tasks we measured, finding code was about a fifth of a full session, which would make the session-wide saving closer to 5–10%. Whole tasks on TokenBench (above) are measured; they run on small repositories, so they test DMN's guidance more than its search.
- Cached tokens count in full. Our token numbers count cached input like any other input. Counting only uncached input and output, the gap shrinks: about 9% on DMN's own code in the September benchmark, and within noise in each of the newer campaigns.
- One agent. Every run used Claude Code: on Claude Opus 5.5 for the newest results, Opus 5 for the September benchmark, Sonnet for the race. Other agents and models may gain more or less.
- Embeddings ran on the CPU in some runs and the GPU in others. That affects speed, not these counts.
Two measures
The app's Insights view and dmn stats compare what DMN handed an agent with reading the same files
whole. That runs higher than the head-to-head numbers on this page, which compare a whole task with DMN to the same
task without it. The head-to-head numbers are the ones we stand behind.
The full report and its protocol are in DMN's repository, which isn't public yet.
Check it on your own code
dmn bench builds a search benchmark from your indexed code, with no labelling, runs it, and writes a
report. The same seed on an unchanged index gives the same numbers. It measures how well DMN finds code, not how
many tokens your agents spend; dmn stats shows that side.
dmn bench