This corpus is the floor. Each record asks one question of one knowledge graph: no cross-graph join, no ontology expansion, no ablation. It exists so the other two studies can be read properly — a model that cannot answer here has not failed at joins or hierarchies, it has failed at plain SPARQL against a schema it was handed.
Every figure here is derived from runs-*-single/conditions/*.json and can be recomputed from summary.csv in the same directories.
Attempts answered exactly, out of 3, one arm (KG named, full toolkit). Exactly here means right rows and right order — ordered_exact, the criterion the figures count and the cross-graph study uses.
| Question (graph) | Opus 5 | Sonnet 5 | Haiku 4.5 | Total |
|---|---|---|---|---|
gene-expression-atlas-okn | 3 | 3 | 3 | 9/9 |
ncipidkg | 3 | 3 | 3 | 9/9 |
oard-kg | 3 | 3 | 3 | 9/9 |
spoke-okn | 3 | 3 | 3 | 9/9 |
digcfdekg | 3 | 3 | 2 | 8/9 |
spoke-genelab | 3 | 3 | 2 | 8/9 |
pankgraph | 3 | 2 | 2 | 7/9 |
biobricks-aopwiki | 3 | 3 | 0 | 6/9 |
biomarkerkg | 3 | 1 | 2 | 6/9 |
nde | 3 | 3 | 0 | 6/9 |
prokn | 3 | 3 | 0 | 6/9 |
rdkg | 3 | 1 | 2 | 6/9 |
| Opus 5 | Sonnet 5 | Haiku 4.5 | |
|---|---|---|---|
| attempts exact | 36/36 | 31/36 | 22/36 |
| records always exact | 12/12 | 9/12 | 4/12 |
| records ever exact | 12/12 | 12/12 | 9/12 |
| mean F1 | 1.0000 | 0.9667 | 0.6663 |
F1 is partial credit over the answer's rows as a multiset: precision = matching rows ÷ rows returned, recall = matching rows ÷ rows in the key, F1 = 2PR/(P+R). A row matches only if it agrees cell for cell, with IRIs compared by local name where unambiguous and decimals at 6 significant digits. Where a table reports a mean, F1 is averaged across a question's attempts first and then across questions. Full definition in SUMMARY.md.
Opus is perfect: 36 of 36, mean F1 1.0000, nothing flaky. This is the single most important number for reading the other two corpora. Every failure Opus records against the cross-graph and ontology corpora is a failure at joining, at expanding a hierarchy, or at species discipline — not at reading a schema and writing a query. The floor holds, so the ceiling is measuring what it claims to.
Sonnet loses five attempts, and they are not distributed noise. Two are biomarkerkg, two are rdkg, one is pankgraph — and the first pair fails differently from the rest. The biomarkerkg misses score F1 1.0 and the figure marks them right rows, wrong order: an ordering failure on a question whose prose states the sort explicitly. The rdkg misses score F1 0.9 and the pankgraph miss 0.0; those are wrong rows. A sort error and a data error are different defects, and only the second means the model misread the graph.
Haiku fails 14 of 36 on questions with no join and no ontology. Three of the twelve records defeat it completely — biobricks-aopwiki, nde and prokn, 0 for 9 between them — and prokn scores 0.0 on all three attempts. This is the finding that reframes its results elsewhere: Haiku's 7/36 on the ontology corpus and 0/18 on that corpus's question arms cannot be read as an ontology-specific weakness, because it is already missing more than a third of the floor.
Four questions separate nobody. gene-expression-atlas-okn, ncipidkg, oard-kg and spoke-okn are 9/9. They are worth keeping precisely because they are easy: a corpus with no question that everyone answers cannot distinguish "this model is weak" from "this record is broken."
| Opus 5 | Sonnet 5 | Haiku 4.5 | |
|---|---|---|---|
| total cost | $11.48 | $4.95 | $3.54 |
| server time | 536s | 633s | 2357s |
| queries run | 201 | 184 | 322 |
Haiku spends 4.4× Opus's server time and runs 60% more queries, for 22 correct answers against 36. The pattern holds across all three corpora: it is cheap per attempt and expensive per correct answer. Note also that no attempt in this corpus hit the agent timeout and none failed to submit — these are clean measurements, unlike the ontology sweep, where several attempts were degraded by upstream 120s timeouts.
No cross-graph join, so nothing here exercises identifier crosswalks, and nothing here can catch the get_join_strategy defect the cross-graph and ontology corpora both surface. No unprompted ontology expansion — nde_single does walk a subclass hierarchy, but its prompt asks for one explicitly, so recognizing the need for a hierarchy is never tested. No ablation, so the value of the toolkit is not measured. Those are the other two studies' jobs; this one establishes only that the models can query a single graph correctly when nothing else is in the way.