Does a model expand an ontology hierarchy when nobody tells it to?

Three models — Opus 5, Sonnet 5, Haiku 4.5 — answering six ontology questions and their six controls over the Proto-OKN federation, one tool condition, three attempts each: 108 episodes.

The question this study asks is narrow and specific. A scientist naming a category — "for cardiovascular disease", "hormone secretion" — does not say "walk the subClassOf closure", because a scientist would not. But the data is annotated at specific terms, so the abstract parent is nearly empty by construction and the question is a subtree query whether or not anyone says so. Recognizing that, unprompted, is the task.

QUESTIONS.md reads the same episodes question by question.

1. Methods

Corpus. 12 .rq files (benchmarks-ontology/ in this repo) across six tiers. Each tier is a pair:

• <kg>-expand_<tier> — the question. Its key requires the subClassOf* closure.
• <kg>-exact_<tier> — the control. Same prompt plus one sentence, "use the term exactly as I have given it — do not expand it to subtypes, descendants or related terms", and its key is the unexpanded answer.

Neither prompt mentions subtypes, descendants, closures or ontologies. The pair differs by exactly that one sentence, so the arm measures obedience against the same question the other arm measures discovery against.

Three ontologies. Four records expand a MONDO disease category, one expands a GO biological process, one expands an HPO phenotype. Two knowledge graphs are joined per record, so the ontology work sits on top of a cross-graph join rather than replacing it.

One arm only. Full toolkit, graph named. Ablations are the cross-graph study's axis; adding them here would confound "did it expand" with "did it have the tools".

Agents. Three attempts per question, driven through claude -p 2.1.243, 900s per-episode deadline, high for Opus, medium for Sonnet, the CLI default for Haiku.

Ground truth. Answer keys executed against the Proto-OKN federation endpoint; scoring at 6 significant digits with a 0.0005 relative tolerance. Execution strictly serial.

How mean F1 is calculated. Partial credit, so a near-miss scores above zero. Per attempt, over the answer's rows as a multiset: precision = matching rows ÷ rows returned, recall = matching rows ÷ rows in the key, F1 = 2PR/(P+R). A row matches only if it agrees cell for cell, with IRIs compared by local name where unambiguous and decimals at 6 significant digits (0.0005 relative tolerance); column names are ignored unless both sides use the same set. Both sides empty scores 1.0. The reported figure averages across a question's attempts first and then across questions, so each question weighs the same.

2. What each attempt produced

The figure orders the columns in pairs — each -expand question immediately followed by its own -exact control — so the two arms of a category can be read together.

Splitting the arms is what makes the result legible. Haiku's combined 7/36 reads as "sometimes succeeds". Split, it is 0/18 on the questions and 7/18 on the controls — every success it has is an arm that told it what to do. That is a categorically different claim, and only the split supports it.

Opus's entire deficit is two records. 18/18 on every control, 15/18 on the questions, losing one attempt on oard-kg-expand_metabolic and two on rdkg-expand_neurological. Nine of twelve records are perfect.

Sonnet's controls are sometimes worse than its questions. It is 3/3 on oard-kg-expand_metabolic and 1/3 on the matching control; 2/3 and 1/3 on autoimmune. A control miss means it expanded when told not to — a compliance failure rather than a capability one, and the opposite failure from Haiku's.

3. Naming the failure, not just counting it

A score says an answer missed. It does not say which mistake produced it, and for a corpus built around specific traps that is the more useful fact. Each tier ships controls — the answer key with exactly one thing broken:

• A — root term only: the closure replaced by the bare term. What a non-expanding agent produces.
• B — keyword matching: a substring test on labels standing in for the hierarchy.
• C, D (hormone secretion): the cross-graph join moved onto a gene symbol, and the same join with the species pinned.

Scoring with --controls controls-ontology matches an answer against those tables by denotation, so an agent's own column names do not defeat it. Six recorded misses carry a named cause:

Only A and B appear, and that is a result rather than a gap. An answer is compared against its own tier's controls — two for every tier, four for hormone secretion — so C and D were tested against the eighteen hormone-secretion answers and matched none of them, for two different reasons.

C — the symbol join — is a trap that was closed before this sweep. It reproduces what an agent gets by joining pankgraph to the expression atlas on the gene symbol instead of the Ensembl accession: the atlas is multi-species and symbols are not, so cow, pig and macaque orthologs pool into a human islet question. Opus produced exactly that table on every attempt of an earlier sweep, because the prompt then left the species to inference. The prompt now says published human expression experiments, and no attempt here reproduces C. A control that stops matching is how you tell a reworded prompt actually fixed something.

D cannot appear in this table at all. It is an equivalence control — a symbol join that also filters the study's organism, which reproduces the answer key exactly, row for row. Matching D means being right, so it can never name the cause of a miss. It is kept because it isolates what the reword bought: the identifier trap and the species trap are separable, and D shows the second one alone is enough. The corpus README calls this out as the only equivalence control in the set.

The -exact hits are the sharpest finding in the study. Both models were told to use the term literally and instead produced the keyword-matching table: they did not merely disobey, they faked the hierarchy with a substring test on the label. A column diagnostic reading "0/3 columns align" cannot express that; a control name can.

4. Where the time goes

The ≥ marks a row containing an attempt killed at the 900s deadline: its server time stopped early and its cost was never reported, so those totals understate — and understate most for the models that struggled most.

This corpus is far more expensive than the single-graph floor: Opus spends 6× the server time here (≥3296s against 536s) for a corpus of the same size. Closure size is what strains the endpoint, and the ontology arm is where it shows.

5. What it cost

Per correct answer the ordering inverts. Haiku is the cheapest model per attempt and the most expensive per answer that was right. Opus is the most expensive per attempt and the cheapest per correct answer.

6. Two findings about the server, not the models

The auto_expand_descendants rewriter emits invalid SPARQL when it collides with the agent's own variable bindings. Observed twice: bind_expansion_to alongside an existing VALUES block produced `Token "?pheno": extraneous input '?pheno', and an agent's own BIND(... AS ?phen) produced *"The target variable ?phen of an AS clause was already used before in the query body"*. Both were returned as ok: true with the error inside the payload — a soft failure that every counter reads as *an agent that chose not to expand*. Runs that need to tell those apart should pass --strict-upstream-errors`.

Where the rewriter does not collide, it expands silently and by default. The same query naming GO_0046879 returned 71 genes with the default and 5 with auto_expand_descendants: false. An exact answer on an -expand arm is therefore not by itself evidence that the agent reasoned about hierarchies — it may only mean the agent left a default alone. This is the most important caveat in the study, and it cuts against the headline result.

get_join_strategy reports can_join: false for pairs that demonstrably join. Confirmed on two independent pairs, one of which joins on byte-identical IRIs.

7. Capability under a resource budget

Section 2 reports accuracy and sections 4–5 report what it cost, as if those were separate facts. They are separate only while nothing is rationed. Under a budget, efficiency is capability — an answer you could not afford is not an answer — and the question needs no new episodes, because what each attempt spent is already recorded. Below, an attempt counts only if it was correct and stayed inside the budget.

This corpus has one condition, so there is no arm to compare: the table is model against budget. It is also by far the most endpoint-expensive of the three corpora — roughly six times the single-graph study's server time for the same number of records — because walking a closure is what strains the endpoint. A server budget therefore bites harder here than anywhere else in the study.

Both sections carry a curve as well as a table. Each line steps up once per correct answer, ordered by what that answer cost, and flattens at the model's uncapped total — so a point reads at this budget per attempt, this model would still have got this many right. The curve is exact at every budget, where the table samples four, and a table of four columns cannot show where two lines cross. There are two curves because there are two scarce resources, and they do not rank the models the same way.

Under a server-time budget

Opus's lead survives rationing of the endpoint. It keeps 21 of 33 at eighty seconds against Sonnet's 16 of 22, and in proportional terms the gap barely moves. What the budget exposes is how much of the ontology work is endpoint work: Opus loses more than a third of its correct answers at ten seconds.

There is no crossing anywhere. From 3.2 seconds upward Opus is strictly ahead of both other models at every budget on the axis — the only figure in the three studies where the uncapped ranking holds at every price. Hold that against the token-budget figure below, drawn from the same episodes, where Opus and Sonnet trade the lead four times and finish tied at ≤$0.50. Two currencies, two different answers to "which model is better", and only the choice of currency separates them.

Under a token budget

Under a token budget the lead vanishes. Uncapped, Opus answers 33 and Sonnet 22 — an eleven-answer gap. At ≤$0.50 they are tied at 15, and at ≤$0.25 Sonnet is marginally ahead (5 against 4). Whatever Opus is buying with its extra tokens, a reader who cannot spend them does not receive it.

The figure shows the tie is not a single crossing but a braided region: Haiku leads to $0.27, Opus from $0.28, Sonnet retakes it from $0.43 to $0.49, and Opus holds it from $0.52 upward with Sonnet drawing level again at $0.69 and $0.75. A table sampling four budgets can land anywhere in that stretch and report a different model as ahead.

Haiku's line is the counter-case, and worth stating plainly: it reaches its ceiling of 7 by $0.30 and is then flat forever, because 7 is all it can answer at any price. A budget cannot make a model capable; it can only reveal which of two capable models is affordable.

This budget was applied after the fact

Every number above is censoring, not a run under a constraint. Each episode was recorded with no budget in force, and a budget is then drawn across what it happened to spend. The assumption fails in a known direction: an agent told it had twenty seconds of endpoint time would triage — fewer exploratory queries, an earlier commit to an answer — and should therefore score better than truncating its unconstrained trajectory predicts. Read the capped columns as a floor and the gaps between models as a ceiling on the effect. What this establishes is that the effect exists and which resource it is denominated in — not its size.

Nothing here is a lower bound in the other direction. It reads the resource use of correct episodes only, and an episode killed at the timeout produced no answer — a failure at every budget, never in a numerator. build_budget.py refuses to print at all if any correct episode has an incomplete record.

One caution specific to this corpus: its failures are overwhelmingly semantic — column misalignment and the control reproductions of section 3 — rather than resource exhaustion. Only two episodes across all three ontology run roots hit the agent timeout, against 25 in the cross-graph study. So the budgets above censor answers that were produced, rather than rescuing ones that ran out of room.

8. Reproducing it

okn-benchmark reference --corpus benchmarks-ontology
okn-benchmark run --corpus benchmarks-ontology --tools all-tools --kg-hint tags \
  --profile profiles/mcp-proto-okn.local.toml --model claude-opus-5 --effort high \
  --repeat 3 --agent-timeout 900 --controls controls-ontology --out runs-opus-ontology
uv run python results-ontology/build_comparison.py
uv run python results-ontology/build_budget.py