Seven figures putting all three models on one pair of axes, plus the scripts that build them. The study is written up in SUMMARY.md, which reads them in order; the per-record detail behind them is in QUESTIONS.md. Both also exist as self-contained HTML — SUMMARY.html and QUESTIONS.html, with the figures embedded and nothing to resolve — rebuilt by uv run python results-ontology/build_html.py.
The corpus itself is benchmarks-ontology/, whose README describes the twelve records and the defects worth knowing before reading any result. The controls the scorer matches against are in controls-ontology/.
| Figure | Shows |
|---|---|
outcome-by-model.svg | what each attempt produced, over all twelve records |
outcome-by-model-unprompted.svg | the same, over the six questions only — the -exact controls dropped |
server-time-by-model.svg | seconds each attempt spent inside the server under test |
queries-by-model.svg | how many queries each attempt ran |
cost-by-model.svg | what each attempt cost, in US dollars |
budget-by-server-time.svg | how many answers each model could still afford at a given endpoint budget |
budget-by-cost.svg | the same, with tokens as the scarce resource |
This is the only corpus whose records come in pairs: each of six questions has an -exact control that repeats the same prompt plus one sentence — "use the term exactly as I have given it — do not expand it to subtypes, descendants or related terms." The question measures discovery; the control measures obedience. They are different skills and a model can be good at one and bad at the other.
outcome-by-model shows all twelve records, which is the honest full picture. But a headline figure that pools them reads as twelve questions rather than six questions with a calibration arm, and that pooling hides the study's main result: Haiku's combined 7/36 reads as "sometimes succeeds", while split it is 0/18 on the questions and 7/18 on the controls — every success it has is an arm that told it what to do. outcome-by-model-unprompted drops the controls so that claim is visible without arithmetic.
Read the pooled figure to audit, the unprompted one to conclude.
The five grids share one layout, so any two can be compared square for square. The two budget figures are a different idiom and are described below.
Rows are the three models, one band each, labeled with the reasoning effort it ran at underneath its name. Unlike the cross-graph study, there is no second level to the banding: this corpus has a single tool condition, so each model contributes one row rather than three. The -expand / -exact split is a column distinction here, not an arm.
Columns are the twelve records, three squares each: one per attempt, left to right. They are ordered so each question sits immediately left of its own control — nde-expand_autoimmune then nde-exact_autoimmune, and so on — so a pair can be read without hunting across the figure.
Blue on an -exact column does not mean the same thing as blue on an -expand column. Five of the six controls key to zero rows, so blue there means the agent correctly answered "nothing satisfies the question" and gray means it invented rows. Only pankgraph-exact_hormonesecretion keys to a non-empty table — two rows — where a broken agent scores 0 like any other wrong answer. Read a blue control cell as "obeyed the instruction", not as "returned nothing".
That asymmetry is a known weakness of the empty-keyed controls and is documented in the corpus README: an empty key scores a perfect 1.0 for any agent that returns nothing, including one that crashed or gave up, so those five columns cannot tell obedience from breakage on their own.
Shading on the three quantitative figures is logarithmic and diverging about the median attempt: blue is below typical, red above. The neutral midpoint is the median of the whole figure and the legend states its value.
A heavy outline marks an attempt killed at the 900s agent deadline. Two attempts here were — Opus on rdkg-expand_neurological and Sonnet on nde-expand_autoimmune — and on outcome-by-model they are drawn hollow instead, because an episode that never reached a verdict has no outcome to color. On cost-by-model they are blank: the agent host writes agent.json only on a clean exit, so no price was ever reported. Blank rather than shaded at the pale end, which would read as "cheap" when it means "unknown, and certainly not free".
The right-hand column totals each row over all 36 of its attempts. A total marked ≥ contains a killed attempt and therefore understates. On outcome-by-model the column is a tally of exact attempts instead — 33/36, 22/36, 7/36 — and carries no ≥: a killed attempt is definitively not exact, so that count is exact even where the sums are only bounds.
Every square has a tooltip naming the model, record and value.
The row totals here are sums over all 36 attempts of a model. The Server and Queries columns in each root's generated README.md are one pass — the per-record mean summed over twelve records — so they are roughly a third of these. Neither is wrong; they answer different questions. Do not put them in the same table.
The grids put one square per attempt, which cannot show a sweep: a budget is a continuous axis, not a condition. The two budget-by-* figures are that other view. Each line steps up once per correct answer, ordered by what that answer cost, and flattens at the model's uncapped total — so a point reads at this budget per attempt, this model would still have got this many right. The curve is exact at every budget, where §7 of SUMMARY.md tabulates four.
There are two because there are two scarce resources, and here they give opposite answers to "which model is better":
Same episodes, opposite conclusions; only the choice of currency separates them. Haiku's line is the counter-case in both: it reaches its ceiling of 7 early and is then flat forever, because 7 is all it can answer at any price. A budget cannot make a model capable; it can only reveal which of two capable models is affordable.
Because this corpus has one condition, these are single-axis curves rather than the small multiples the cross-graph study needs — there is no arm dimension to split on.
Each figure exists as .svg and .png. The SVG is the source of truth — sharper at any zoom, a fifth of the size, and what build_comparison.py writes. The PNGs are rendered from it at 2× (2000px wide) for venues that will not take SVG: manuscript submission, Word, PowerPoint. SUMMARY.md links the PNGs so it stays portable into those tools; the table above links the SVGs.
All seven are hand-built. The harness renders one runs root at a time and cannot draw three models on one pair of axes, so nothing regenerates them automatically:
uv run python results-ontology/build_comparison.pyThe script reads the three roots directly and writes all seven figures. To refresh the PNGs afterwards:
./results-ontology/build_png.shIt renders each SVG through headless Chrome — the same engine the figures were designed against — so the PNG shows what a reader sees in the browser. Set SCALE=3 for a higher-DPI render. Re-run it after any run, rescore or report that changes a root, or the figures will quietly disagree with the tables they sit beside.
It shares its palette, ramp, layout and timeout conventions with the harness's own figure code (src/okn_benchmark/figure.py), which is what keeps a comparison figure readable straight after a per-root one — and what keeps this study's figures readable straight after the other two corpora's.
build_budget.pyRe-reads the recorded episodes under a resource budget and writes the §7 tables to stdout as markdown. It needs no model, no endpoint and no network: the budgets are applied to runs that already exist, so an episode counts only if it was correct and stayed inside the cap.
uv run python results-ontology/build_budget.py
It refuses to print if any correct episode is missing its recorded cost. The section's claim that these numbers are not lower bounds rests on timeout-killed episodes producing no answer and so never entering a numerator; a silent hole would look exactly like a result. Both of this corpus's killed attempts submitted nothing, so neither can enter a numerator at any budget.
The tables here are model × budget with no arm dimension, because this corpus has a single condition. The computation lives in okn_benchmark.budget and is tested; this script only declares the table shape.