The single-graph floor: can three models query one graph correctly?

Three models — Opus 5, Sonnet 5, Haiku 4.5 — answering the same twelve single-graph questions over the Proto-OKN federation, one tool condition, three attempts each: 108 episodes.

This study is a baseline. Its job is to establish that the models can read a schema and write a correct query against one knowledge graph, so that failures in the cross-graph and ontology studies can be attributed to joins and hierarchies rather than to basic competence. A ceiling measurement is only meaningful against a floor.

QUESTIONS.md reads the same episodes question by question.

1. Methods

Corpus. 12 .rq files (benchmarks-single/ in this repo), all on the single tier — one per knowledge graph. Each pairs a prose #+ summary: — the prompt given to the agent — with a SPARQL answer key. The summary states every limit and tie-breaker, so the result set is fully determined by the prose. It never names the knowledge graph; choosing it is part of the task, though the --kg-hint tags arm used here does supply it.

Knowledge graphs. One per question, twelve in all: biobricks-aopwiki, biomarkerkg, digcfdekg, gene-expression-atlas-okn, ncipidkg, nde, oard-kg, pankgraph, prokn, rdkg, spoke-genelab, spoke-okn. One record reads a thirteenth: nde_single also walks rdfs:subClassOf* in ubergraph, because its question counts a disease together with its subclasses. The prompt says so outright, so no expansion is being inferred — but the ontology path is exercised here, and get_descendants is called in all three sweeps.

One arm only. The full toolkit with the graph named (all-tools, --kg-hint tags). There is no ablation here: withholding tools measures what the tooling is worth, which is the cross-graph study's question. Adding that axis to a floor measurement would only make the floor harder to read.

Agents. Three attempts per question, driven through claude -p 2.1.243, with a 900s per-episode deadline. Each model ran at its own reasoning effort — high for Opus, medium for Sonnet, and the CLI default for Haiku, which takes no --effort flag.

Ground truth. Answer keys are executed against the Proto-OKN federation endpoint (https://apps.okn.us/federation/sparql); the rows they return are the reference. Scoring compares at 6 significant digits with a 0.0005 relative tolerance.

How mean F1 is calculated. Partial credit, so a near-miss scores above zero. Per attempt, over the answer's rows as a multiset: precision = matching rows ÷ rows returned, recall = matching rows ÷ rows in the key, F1 = 2PR/(P+R). A row matches only if it agrees cell for cell, with IRIs compared by local name where unambiguous and decimals at 6 significant digits (0.0005 relative tolerance); column names are ignored unless both sides use the same set. Both sides empty scores 1.0. The reported figure averages across a question's attempts first and then across questions, so each question weighs the same.

Execution. Strictly serial — timings are meaningless under contention.

2. What each attempt produced

Attempts exact, below, is ordered_exact — right rows and right order, the stricter of the two criteria and the one the figure's totals count, matching the cross-graph study. The order-insensitive exact is two higher for Sonnet (33) and identical for the other two models; the two attempts that separate them are biomarkerkg, and they are discussed below.

Opus is perfect, and that is the study's main result. 36 of 36, mean F1 1.0000, nothing flaky, no attempt timed out and no attempt failed to submit. Every Opus failure recorded in the other two corpora is therefore a failure at something those corpora add — a cross-graph join, an ontology closure, a species constraint — and not at querying a graph.

Sonnet loses five attempts, and they fail in two different ways. Two are biomarkerkg, scoring F1 1.0 and appearing in the figure as right rows, wrong order — every row correct and the sort not, on a question whose prose states the sort explicitly. The other three are genuinely wrong rows: two on rdkg at F1 0.9, a near miss that is still the wrong answer, and one on pankgraph at F1 0.0. Keeping those two states apart is what the figure's four-entry legend is for: a sort error and a data error are not the same defect, and only one of them means the model misread the graph.

Haiku misses 14 of 36 with nothing in the way. Three records defeat it entirely — biobricks-aopwiki, nde and prokn, 0 for 9 between them. Its ever exact of 9/12 against always exact of 4/12 says the remaining failures are instability rather than incapacity: it can reach five more records, just not reliably.

3. Where the time goes

Haiku spends 4.4× Opus's server time for 22 correct answers against 36. This is the clearest statement of the pattern that recurs in all three studies: cheap per attempt, expensive per correct answer. Sonnet spends 18% more than Opus for five fewer correct answers.

No total here carries a ≥ marker, because no attempt hit the deadline — unlike the ontology sweep, where two attempts were killed mid-flight and their totals understate.

4. How much work reached the endpoint

Sonnet answers 31 of 36 in fewer queries than Opus needs for 36. Query count is not a proxy for competence: Haiku runs 60% more queries than Opus and answers 14 fewer, while Sonnet runs 8% fewer than Opus and answers 5 fewer. What separates the models is whether the queries were the right ones.

5. Which tools were used

All three call get_schema about three times per record, which is the intended workflow: read the schema before writing SPARQL. The discovery tools beyond that are barely touched, which is expected — a single-graph question has nothing to discover across graphs. get_join_strategy is called once in 108 episodes, on a corpus with no joins.

Haiku submits 34 CSVs across 36 attempts. Two attempts produced no answer at all.

6. What it cost

Per correct answer, Sonnet and Haiku are indistinguishable — and Haiku, nominally the cheapest per token, is no cheaper than Sonnet once you divide by answers that were right. Opus costs twice as much per correct answer and is the only model that never got one wrong.

7. Capability under a resource budget

Section 2 reports accuracy and sections 3–6 report what it cost, as if those were separate facts. They are separate only while nothing is rationed — the single ceiling here was the 900s agent timeout, which bound on no episode in this corpus at all. Under a budget, efficiency is capability: an answer you could not afford is not an answer. The question needs no new episodes, because what each attempt spent is already recorded. Below, an attempt counts only if it was correct and stayed inside the budget.

This corpus has one condition, so there is no arm to compare — the table is model against budget. That makes it the cleanest place in the study to read the effect, because nothing else varies.

Both sections carry a curve as well as a table. Each line steps up once per correct answer, ordered by what that answer cost, and flattens at the model's uncapped total — so a point reads at this budget per attempt, this model would still have got this many right. The curve is exact at every budget, where the table samples four, and a table of four columns cannot show where two lines cross. There are two curves because there are two scarce resources, and they do not rank the models the same way.

Under a server-time budget

Rationing the endpoint costs every model something and changes little. Opus keeps 24 of its 36 at ten seconds and Haiku keeps 12 of its 22 — both lose about a third, and the ranking at every one of these four caps is the ranking it already had.

At finer resolution the ordering is not quite as stable as the columns suggest. Sonnet leads Opus below 6.8 seconds, by as many as three answers around the 3-second mark, and the two braid across several crossings before Opus takes the lead outright and keeps it. Haiku leads at no budget at all. The braid is small — three answers out of thirty-six — and it is the honest reading of a figure that shows every crossing rather than four samples that happen to miss them all.

Compare this with the token budget below, where the same three models reorder completely. That contrast is the point of showing both: endpoint time and tokens are different currencies, and a model cheap in one can be dear in the other.

Under a token budget

Here the ordering inverts, and it is the sharpest result in this study. At ≤$0.10 Opus scores zero and Haiku scores 17. A perfect 36/36 is worth nothing when tokens are the scarce resource: every one of Opus's correct answers cost more than a dime, and Haiku's cost less. Opus does not overtake Sonnet until $0.41.

The figure shows what the four columns cannot: Opus is flat on zero until $0.11. Its problem is not that it is gradually expensive — no correct answer it produced cost less than that, so below that price it returns nothing at all. Above it the line climbs almost vertically. Haiku is the mirror image: it rises earliest, reaches 22 by $0.22, and is then dead flat forever, because 22 is all it can answer at any price.

Sonnet leads Opus from $0.14 to $0.38 — most of the interval a reader would call a tight budget — and the handover is not a single crossing but a short braided stretch between $0.38 and $0.41 where the two trade places. Opus holds the lead only from $0.41 upward.

The model that wins outright with no budget is the worst model at a tight one. That is not a statement about capability in the ordinary sense — it is what capability means once a resource is scarce.

This budget was applied after the fact

Every number above is censoring, not a run under a constraint. Each episode was recorded with no budget in force, and a budget is then drawn across what it happened to spend. The assumption fails in a known direction: an agent told it had twenty seconds of endpoint time would triage — fewer exploratory queries, an earlier commit to an answer — and should therefore score better than truncating its unconstrained trajectory predicts. Read the capped columns as a floor and the gaps between models as a ceiling on the effect. What this establishes is that the effect exists and which resource it is denominated in — not its size.

Nothing here is a lower bound in the other direction. It reads the resource use of correct episodes only, and an episode killed at the timeout produced no answer — a failure at every budget, never in a numerator. build_budget.py refuses to print at all if any correct episode has an incomplete record.

Note also that failures in this corpus are overwhelmingly semantic rather than resource exhaustion: no episode here hit the agent timeout, so nothing was truncated in the recorded runs and the censoring above is the only budget any episode ever met.

8. What this study does not measure

No cross-graph join, so identifier crosswalks are never exercised and the get_join_strategy defect the other two corpora surface cannot appear here. No unprompted ontology expansion: nde_single walks a subclass hierarchy, but its prompt asks for one in so many words, so nothing here tests whether a model recognizes that a question needs a hierarchy. No ablation, so the value of the toolkit itself is not measured.

Those omissions are the point. This corpus answers one question — can the model query a single graph correctly — and the answer is yes for Opus, nearly always for Sonnet, and often not for Haiku. Every richer claim belongs to the studies that add a variable.

9. Reproducing it

okn-benchmark reference --corpus benchmarks-single
okn-benchmark run --corpus benchmarks-single --tools all-tools --kg-hint tags \
  --profile profiles/mcp-proto-okn.local.toml --model claude-opus-5 --effort high \
  --repeat 3 --agent-timeout 900 --out runs-opus-single
uv run python results-single/build_comparison.py
uv run python results-single/build_budget.py