This corpus asks whether a model works out, unprompted, that a question needs an ontology hierarchy. Each of six categories is a pair: an -expand arm whose answer requires walking the subClassOf* closure, and an -exact arm whose prompt adds one sentence — "use the term exactly as I have given it" — and whose key is the unexpanded answer. Neither prompt mentions subtypes, descendants or ontologies; recognizing that the data is annotated at specific terms is the task.
Every figure here is derived from runs-*-ontology/conditions/*.json and can be recomputed from summary.csv in the same directories.
Attempts answered exactly, out of 3, as expand / exact:
| Category (ontology) | Opus 5 | Sonnet 5 | Haiku 4.5 | Total |
|---|---|---|---|---|
| cardiovascular (MONDO) | 3/3 | 3/3 | 0/1 | 13/18 |
| hormone secretion (GO) | 3/3 | 1/3 | 0/1 | 11/18 |
| metabolic (MONDO) | 2/3 | 3/1 | 0/2 | 11/18 |
| neurological (HPO) | 1/3 | 0/3 | 0/3 | 10/18 |
| autoimmune (MONDO) | 3/3 | 2/1 | 0/0 | 9/18 |
| musculoskeletal (MONDO) | 3/3 | 1/1 | 0/0 | 8/18 |
| Opus 5 | Sonnet 5 | Haiku 4.5 | |
|---|---|---|---|
-expand arms (the question) | 15/18 | 10/18 | 0/18 |
-exact arms (the control) | 18/18 | 12/18 | 7/18 |
| all attempts | 33/36 | 22/36 | 7/36 |
| records always exact | 10/12 | 5/12 | 1/12 |
| mean F1 | 0.9685 | 0.6778 | 0.2233 |
F1 is partial credit over the answer's rows as a multiset: precision = matching rows ÷ rows returned, recall = matching rows ÷ rows in the key, F1 = 2PR/(P+R). A row matches only if it agrees cell for cell, with IRIs compared by local name where unambiguous and decimals at 6 significant digits. Where a table reports a mean, F1 is averaged across a question's attempts first and then across questions. Full definition in SUMMARY.md.
Haiku scores 0 of 18 on the question arms, and every one of its 7 successes is a control. The combined 7/36 reads as "sometimes succeeds"; split by arm it is "never expands, sometimes obeys." Those are different claims, and only the split supports one. Read this beside the single-graph corpus, where Haiku already misses 14 of 36 with no ontology involved — its failure here is not specific to hierarchies.
Opus's entire deficit is two records. It is 18/18 on every control and 15/18 on the questions, losing one attempt on oard-kg-expand_metabolic and two on rdkg-expand_neurological. Nine of the twelve records are perfect. This is not a model that is unreliable across the corpus; it is a model that is completely reliable except on one hard record.
The neurological pair is the widest split in the corpus: 1/9 on expand, 9/9 on exact. That is by design. rdkg annotates HP_0000707 zero times directly, so obeying "use the term exactly" correctly returns the empty table — trivial to get right — while the question arm requires reaching 459 phenotypes through the closure. All three models obey; only Opus ever expands.
Sonnet is the only model whose controls are worse than its questions on some categories. It is 3/3 on oard-kg-expand_metabolic but 1/3 on the matching control, and 2/3 versus 1/3 on autoimmune. A control miss means it expanded when told not to — a compliance failure, not a capability one.
The corpus ships a control per known mistake: the answer key with exactly one thing broken. An answer that reproduces a control's table is diagnosable rather than merely wrong, and the scorer records the match (--controls controls-ontology). Six recorded misses carry a named cause:
| Model | Record | Reproduces |
|---|---|---|
| Sonnet 5 | nde-exact_autoimmune (×2) | autoimmune/B-keyword-matching |
| Sonnet 5 | pankgraph-expand_hormonesecretion | hormonesecretion/A-root-term-only |
| Haiku 4.5 | nde-exact_autoimmune (×2) | autoimmune/B-keyword-matching |
| Haiku 4.5 | nde-expand_autoimmune | autoimmune/B-keyword-matching |
The -exact hits are the sharpest result here. Both models were told to use the term literally and instead produced the keyword-matching table: they did not merely disobey, they faked the hierarchy with a substring test on the label. A column diagnostic saying "0/3 columns align" cannot express that; a control name can.
| Opus 5 | Sonnet 5 | Haiku 4.5 | |
|---|---|---|---|
| total cost | ≥$26.73 | ≥$22.34 | $8.38 |
| server time | ≥3296s | ≥3681s | 4165s |
| queries run | ≥457 | ≥595 | 502 |
The ≥ marks rows containing an attempt killed at the 900s deadline: its server time stopped early and its cost was never reported, so those totals understate — and they understate most for the models that struggled most. Haiku again spends the most server time for the fewest correct answers.
Several attempts were degraded by the endpoint, not by the model. The recorded sweep includes attempts that lost 120s per call to upstream timeouts, and one Opus attempt on rdkg-expand_neurological hit the 900s agent deadline with no answer submitted. Where an attempt died on the plumbing, the score is a floor rather than a measurement.
The server can expand without being asked. Where the query tool's rewriter does not collide with the agent's own bindings, auto_expand_descendants expands silently and by default — the same query returned 71 genes with the default and 5 with it disabled. An exact answer on an -expand arm is therefore not by itself evidence that the agent reasoned about hierarchies. The benchmarks-ontology/README.md records this, along with the rewriter defect that makes the converse fail: colliding bindings produce invalid SPARQL returned as ok: true, which every counter reads as an agent that chose not to expand.