benchmarks-ontology — cross-KG questions that ontology expansion decides

Benchmark records kept out of benchmarks-cross/ so they do not disturb the frozen 12-question cross-graph corpus behind results-cross/SUMMARY.md.

Start with nde-expand_autoimmune.rq. It tests the same skill as the cardiovascular record for a sixth of the closure (259 concepts against 1,592) and a sharper keyword failure, and it is the one verified end-to-end through the harness — an exact, ordered match in 156s and 19.9s of server time. See Choosing between the two.

It takes the published cardiovascular-disease expansion walkthrough — a single-graph question against the NIAID dataset catalog — and carries it across a second graph, where the expansion stops being a recall improvement and becomes the difference between an answer and no answer.

What it tests: whether the server works out, unprompted, that the question needs an ontology hierarchy. The prompt is a scientist's request. It says "for cardiovascular disease" and nothing about subtypes, descendants, closures or MONDO — because a scientist would not. Recognizing that the data is tagged at specific conditions, and that the question is therefore a subtree query, is the task.

The corpus holds twelve records: six questions, each with a compliance control. Every question asks something a scientist would ask; every control repeats that prompt with one sentence added — "use the term exactly as I have given it" — and keys to what obeying produces, so obeying scores 1.

Three ontologies are covered. Four records expand a MONDO disease category, one expands an HPO phenotype, and one expands a GO biological process. The skill is the same; the ontologies, the KG pairs and the shape of the failure are not.

The HPO tier. rdkg-expand_neurological uses HP_0000707, which rdkg annotates zero times directly while its closure reaches 459 phenotypes that rdkg does use. The term used literally therefore matches nothing, and a non-expanding agent returns the empty table rather than a plausible subset — the failure is unmissable rather than partial-credit.

A caution for anyone adding a record. A graph's advertised identifier_namespaces are frequently crosswalk annotations rather than primary identifiers. Measure the density of the identifier you intend to join on before designing the record: one candidate graph advertised HPO and carried it on about 1% of its traits, which is not enough to key an answer against.

A defect in the server's expansion tooling, worth knowing before reading any result. The query tool's auto_expand_descendants rewriter emits invalid SPARQL when it collides with the agent's own variable bindings. Two observed forms:

• with bind_expansion_to: ["pheno"] alongside an existing VALUES ?pheno { ... }, the endpoint rejects the rewritten query with `Token "?pheno": extraneous input '?pheno' — a variable has been injected into the data section of a VALUES` block, where only IRIs and literals are legal;
• with an agent's own BIND(<HP_0000855> AS ?phen), the rewrite produces "The target variable ?phen of an AS clause was already used before in the query body".

Both are returned as ok: true with the error inside the payload. A soft failure like that reads to every counter as an agent that chose not to expand, so a record can score an agent 0 for a defect in the server it is being asked to use. Runs that need to tell those apart should pass --strict-upstream-errors.

The converse is worth knowing too: where the rewriter does NOT collide, it expands silently and by default. The same query naming GO_0046879 returned 71 genes with the default and 5 with auto_expand_descendants: false. An agent can therefore obtain a closure without ever reasoning about hierarchies, which is a caution against reading any expansion result as evidence of agent reasoning.

Controls C and D are worth reading together. controls-ontology/hormonesecretion/C-symbol-join.rq documents a failure an agent actually made: joining pankgraph to the expression atlas on the gene SYMBOL rather than the Ensembl accession. The atlas is multi-species and symbols are not, so the join silently pools cow, pig and macaque orthologs into a human islet question. Under the prose in force until 2026-08-24 this inflated three counts and pushed a real gene out of the top 15 in favour of one that only qualifies on pig data — and Opus produced exactly that table on every recorded attempt.

That prose never said "human"; the species was left to inference, against the corpus rule that a summary states every limit. It now reads "published human expression experiments", and controls-ontology/hormonesecretion/D-symbol-join-human-filtered.rq records what that cost: a symbol join that also filters the study's organism reproduces the key exactly. D is an equivalence control, not a defect — the only one in the corpus. The record still tests ontology expansion and now tests species discipline; it no longer isolates the identifier trap, because an agent can be right without ever finding it. C remains a clean diagnostic for a symbol join with no species filter.

Naming. Each pair is <kg>-expand_<tier> and <kg>-exact_<tier>, echoing the one sentence that separates them: the -exact arm adds "use the term exactly as I have given it — do not expand it to subtypes, descendants or related terms." Both arms use the ontology; only -expand walks it.

Two kinds of control key. Five controls key to zero rows and therefore need #+ expect-empty: true. That is normally a weakness: an empty key scores a perfect 1.0 for any agent that returns nothing, including one that crashed or gave up, so the arm cannot tell obedience from breakage on its own. One control — hormone secretion, two rows — keys to a non-empty table, where a broken agent scores 0 like any other wrong answer. Prefer the non-empty form where the data allows it.

rdkg-exact_neurological is the deliberate exception: rdkg never annotates HP_0000707 directly, so an empty table is the only honest answer to "use the term exactly as I have given it", and no rewording produces a non-empty one. That same absence is what makes its paired question sharp — a non-expanding agent returns nothing at all rather than a plausible subset.

The recorded three-model sweep covers all twelve records. runs-opus-ontology/, runs-sonnet-ontology/ and runs-haiku-ontology/ each hold three attempts on every record, including the HPO and GO tiers, and the figures in results-ontology/ are built from all twelve by results-ontology/build_comparison.py.

Each record is a prose #+ summary: (the prompt), #+ tags: (the KG hint), and an answer key. Control A is the key with the category term used literally; control B is the key with a name test standing in for the hierarchy. Both are plain SPARQL, not benchmark records — corpus list does not descend into controls-ontology/.

What the expansion is worth is a property of the keys, not a measurement. For each category it is the distance between the question's key and its control's — fifteen rows against zero — fixed before any agent runs. The corpus does not spend episodes re-deriving it. What does need a run is whether the agent obeys when told not to expand, and that is what the -exact arm scores: it separates the models (Opus 12/12, Sonnet 6/12, Haiku 3/12 across the four MONDO controls) where the questions alone do not.

The cardiovascular record, in full

The sections below walk through nde-expand_cardiovascular — the original, and the one whose controls are documented in most detail. nde-expand_autoimmune is the same design over a smaller category; Choosing between the two tabulates the differences.

The question

I want to find cardiovascular conditions where I could pair open research data with real-world clinical evidence. For cardiovascular disease, give me the conditions that have both catalogued research datasets and electronic-health-record-derived associations to phenotypic features. Report the condition's name, its IRI and the number of distinct datasets. Give me the top 15 by dataset count, breaking ties by condition IRI.

Two graphs are tagged, and the hierarchy comes from a third the server reaches on its own:

ubergraph is deliberately absent from #+ tags:. The tags become the --kg-hint tags line in the prompt, and naming the ontology hub there would hand over the answer: it is the one graph whose presence in the query is the finding. Note that the condition names come from nde, not from the ontology — so the hierarchy is the hub's only contribution, and an episode that skips it is missing exactly one thing.

Both graphs key diseases on the same raw http://purl.obolibrary.org/obo/MONDO_{id} IRI, so the cross-graph join needs no rewrite. All the difficulty is in choosing the conditions.

The answer

Verified against https://apps.okn.us/federation/sparql on 2026-08-21 (0.5s); selfcheck passes exact and ordered.

The 15th and 16th rows are a genuine tie at 33 datasets — mucocutaneous lymph node syndrome and angiosarcoma (0016982) — so the cut is decided by the stated tie-break rather than by whatever order the engine happened to return. See Why ties break on the IRI.

81 conditions qualify in all; the record reports the top 15. The full expansion behind them reproduces the published walkthrough exactly: 1,592 concepts in the closure, 284 of them actually used to tag datasets, 10,324 dataset–condition pairs.

Why there is no phenotype column

There was one, and it earned nothing. The prompt used to ask for a fourth column — the number of distinct phenotypic features — and to break ties on it before falling back to the name. Dropping both leaves oard-kg in the query as an existence test (?assoc bl:subject ?condition, with no bl:object), which is what the question actually asks: conditions that have EHR evidence, not how much of it they have.

The reduction is free. The same 15 conditions come back in the same order, A still returns 0 rows and B still returns its same 12 with the same 4 correct — because the discriminating property is that the root term has zero EHR associations, and that is true whether or not they are counted. What changes is the work: one join pattern fewer, one column fewer to carry through a GROUP BY, two tie-break levels fewer to get right, and a key that runs in ~0.5s instead of ~0.9s.

What the expansion is worth

Run the identical query — same dataset join, same EHR join, same counts, same ordering, same LIMIT — and change only how the conditions are chosen:

A — why crossing graphs is what makes this bite

This is the interesting failure, and it is not the same as the walkthrough's. MONDO_0004995 is not a dead end on the dataset side: 447 datasets are tagged at the root term itself. The single-graph question therefore answers without any expansion, returns a plausible number, and gives no hint that 9,877 further dataset–condition pairs exist.

The EHR graph is different. It records zero phenotype associations for the root, because clinical records are coded at specific conditions and never at the abstract category. So the join has nothing to meet on and the entire table vanishes. Crossing to a second graph converts a quietly incomplete answer into no answer at all — which is why this makes a better test than the single-graph version it came from.

B — why keyword matching is worse than it looks

It returns a well-formed 12-row table that a reader has no way to tell is wrong.

• It keeps 4 of the true top 15 — the four spelled with "cardio".
• It misses 11, including the 2nd through 6th largest: preeclampsia (144 datasets), multifocal atrial tachycardia (143), Kaposi's sarcoma (123), pulmonary arterial hypertension (100), myocarditis (70), eclampsia (61), vasculitis (50), aortic valve stenosis (35), angiosarcoma (33). No cardiovascular word appears in any of those names; the ontology is the only thing that knows they are cardiovascular conditions.
• It invents one: familial avascular necrosis of femoral head, a bone disease, because "a-vascular" contains "vascular". Substring matching has no idea what a word means, let alone what a class is.

Only C is right, and the hierarchy is what makes it right in both directions: it reaches the conditions that are not named cardiovascular, and it excludes the one that only sounds it.

Running it

uv run okn-benchmark corpus list --corpus benchmarks-ontology
uv run okn-benchmark reference --corpus benchmarks-ontology     # 15 rows, ~0.5s, no model
uv run okn-benchmark selfcheck --corpus benchmarks-ontology     # must pass

uv run okn-benchmark run --corpus benchmarks-ontology --tools all-tools \
    --profile profiles/mcp-proto-okn.local.toml \
    --kg-hint tags --model claude-opus-5 --effort high --repeat 3 --out runs-opus-ontology

There is no ontology tier. tier_of() takes the text after a record's last underscore, so this corpus's tiers are its four categories — autoimmune, cardiovascular, metabolic, musculoskeletal — and --tier ontology matches nothing. The recorded sweeps used no --tier at all, which is how every record lands in one root. That is safe only because there is a single condition; with a second, the collision guard would fire.

To run one category and its control:

uv run okn-benchmark run --corpus benchmarks-ontology --tier autoimmune --tools all-tools \
    --profile profiles/mcp-proto-okn.local.toml \
    --kg-hint tags --model claude-opus-5 --effort high --repeat 3 --out runs-opus-ontology

A note on which server build you are measuring

The mcp-proto-okn ontology tools were broken for most of the work behind this corpus, and the failure was severe enough to invalidate whole sweeps: lookup_uri and get_descendants returned HTTP 429 after ~30s, and a single such call took the MCP session down with it. Whole sweeps died that way, scoring 0 for reasons that had nothing to do with the questions.

It was never a rate limit. Both tools query the federation's own ubergraph named graph, and both built queries the endpoint could not serve — get_descendants a UNION of explicit join chains, one per depth, instead of rdfs:subClassOf*; lookup_uri an LCASE() on the stored label, which no index can answer, so it scanned every label in ubergraph twice. The repairs are in the mcp-proto-okn repository:

So a run is only comparable with another run against the same build. Point --profile at profiles/mcp-proto-okn.local.toml to measure a local checkout; the deployed apps.okn.us/okn-mcp is whatever it is serving that day, and was still serving the broken tools at the time of writing. Every result recorded here before the repair is measuring the defect rather than the question — including the ones this README quotes, unless it says otherwise.

Choosing between the two

Both records ask the same shape of question against the same two graphs, and both are decided by the same subClassOf* closure. They differ in the category, and the category is what costs.

The autoimmune record is the better test on the two axes that matter.

It is six times cheaper. Closure size is what strains the server's ontology tooling, and get_descendants is the tool an agent reaches for first.

Its keyword control fails harder. Control B's whole job is to show that a substring on the name is not a substitute for a class. Autoimmune diseases are named after the person who described them or the tissue they attack — systemic lupus erythematosus, Sjogren syndrome, Graves disease, myasthenia gravis, scleroderma — and never after the mechanism, so keyword matching misses the ten largest and returns a table topped by a condition with 20 datasets when the true top row has 594. A thirty-fold error in the headline number. The cardiovascular control B at least catches its own top row.

The cardiovascular record keeps one thing the autoimmune one does not: a control B that invents a row rather than merely missing them — familial avascular necrosis of femoral head, a bone disease, because "a-vascular" contains "vascular". That precision failure is worth keeping as documentation. Keep both records; run the autoimmune one.

Why ties break on the IRI

Both records order by DESC(?datasetCount) ASC(?condition) — count first, then the MONDO IRI. They used to break ties on the condition name first, and that was a bug, caught when a run scored exact=true, ordered=false on an otherwise perfect answer.

The autoimmune table ties at 61 datasets between scleroderma and Sjogren syndrome. The endpoint sorts scleroderma first — a locale-aware collation, where case is a secondary key. The agent sorted Sjogren first — plain codepoint order, where S (0x53) precedes s (0x73). Both are defensible readings of "alphabetically", and they disagree, so an agent could produce the right 15 rows in a right order and still be marked wrong. A benchmark cannot ask for an ordering the engines themselves do not agree on.

MONDO IRIs are ASCII, uniform in shape, and unique, so (datasetCount, IRI) is a total order with exactly one reading. It also shortens the prompt: one tie-break instead of two.

This changed row 15 of the cardiovascular key — angiosarcoma and mucocutaneous lymph node syndrome both have 33 datasets, and MONDO_0012727 sorts before MONDO_0016982.

What else was tried, and why it was rejected

Alternatives were searched by the properties that make this kind of record work, not by domain. A category has to satisfy all four, and most fail on the third:

1. its root carries data in nde, so a single-graph query looks fine; 2. its root carries zero EHR associations in oard-kg, so the cross-graph form collapses; 3. its members are not named after it, or a substring reproduces the answer and the ontology is decorative; 4. enough members qualify to fill a top-15.

The pattern behind (3) is worth stating, because it is the property that makes the ontology load-bearing rather than decorative: the categories that work are mechanism- or inheritance-based, and their members are eponymous. Anatomical and morphological categories name their children after themselves — every leukemia is called a leukemia — so a substring finds them and the hierarchy adds nothing.

The seventh criterion, learned the hard way

A candidate has to satisfy six properties to work as a record here; they are tabulated in What else was tried. A seventh was added after oard-kg_respiratory was built, measured, repaired twice and finally retired:

7. The second graph must model its relationship ONE way.

That record joined oard-kg phenotypes to prokn gene evidence over the HP hierarchy. prokn attaches genes to a phenotype by two different routes, and neither is a subset of the other:

Recurrent pneumonia (HP_0006532)   direct blm:associated_with   89 genes,  via prokn node   0
Atelectasis         (HP_0100750)   direct blm:associated_with    0 genes,  via prokn node  72

The second route goes through prokn's own phenotype node, which points at the HP term with skos:exactMatch. Both are real gene-phenotype evidence.

Every agent that attempted the question picked ONE route and answered consistently within it. None used both, and nothing in the data announces that a choice exists. The key cannot resolve this by picking a side — that is an undisclosed decision, and it produced two wrong answer keys before it was noticed. Nor can it be resolved by unioning both: agents then top out at f1≈0.93 (partial credit over rows; defined in the root README.md), missing exactly the phenotypes modelled the way they did not choose. A benchmark whose ceiling is not 1.0 for a competent approach will mislead whoever reads it.

The record also could not have the collapse property the others rely on. Every HP body-system category carries gene evidence at the root — 1,200 to 2,800 genes — so control A returned 1 row rather than 0, and no choice of category fixed it.

Check criterion 7 before writing anything: probe the second graph's relationship for a term that has it, and confirm there is exactly one path from the partner entity to the ontology term.

A caveat on the first sweep: the 900s timeouts were the plumbing

The first three-attempt sweep of nde-expand_cardiovascular looked flaky — exact_rate 0.33 with the full toolkit against 0.67 with get_descendants withheld, i.e. the ablated arm apparently beating the complete one. Most of that gap is not about the question.

Of the ten attempts recorded across runs-opus-ontology-cv/ and runs-opus-ontology-prompt/, four hit the 900s agent timeout. Three of those four end on a get_descendants call that returned, after ~31s:

{"uri": "…/MONDO_0004995", "label": "cardiovascular disorder",
 "descendant_count": 0, "descendants": [],
 "error": "Failed to fetch descendants: HTTP Error 429: Too Many Requests"}

Ubergraph rate-limits the call; the tool answers isError=false and puts the failure in the body. Two of those attempts also record UpstreamError: could not reach https://apps.okn.us/okn-mcp/mcp. The episodes then make no further tool call at all — 865s of a 900s budget spent waiting on a reply that was never coming, against 32–47s of actual server time.

It is not a size problem, which matters because it rules out fixing it by choosing a smaller question. lookup_uri with max_results: 10 fails the same way, and so does get_descendants on a 259-concept closure. Every Ubergraph-backed tool on the server is capped, whatever it is asked for. The federation SPARQL endpoint is unaffected — the answer keys here all run in under two seconds — so an agent that expands with query + subClassOf* gets through and one that reaches for the dedicated ontology tool does not.

Three things in the harness were wrong. All three are fixed, and the fixes were confirmed against the live server while it was in exactly this failing state:

• Forwarded calls had no read timeout. Unbounded is the MCP client's default, so a server that accepts tools/call and loses the response blocks the session with no way out. Every forwarded call now carries the proxy's --timeout (120s default) as a read timeout.
• The proxy died with its upstream. A second get_descendants hung, hit the new 120s bound, and the streamable-HTTP session went down with it — which used to end the proxy process, taking submit_csv with it even though that tool is served locally and needs no upstream at all. The connection is now held in its own task beside the stdio server, so losing it marks the session and is recorded as lost_upstream in metrics (kept distinct from startup_error: an episode that never had tools is not the same as one that lost them part-way).
• A dead session read as a transient one. With the proxy alive but the connection gone, one observed episode retried into it 86 times and submitted nothing. The reply now says the connection will not come back, that retrying is pointless, and that submit_csv still works. Retries after the loss fell to 5 and 17 in the next two attempts, and both episodes ended on their own — 254s and 262s against a 300s budget — instead of being cut off by the timeout.

Separately, calls.json now records a soft_error per call and an upstream_soft_errors tally, so "the server broke and said so quietly" stops reading as "the agent chose not to expand". --strict-upstream-errors additionally relabels such a reply as an error before the agent sees it; the payload is forwarded verbatim either way, only the flag differs. It is off by default, because a server that fails softly is a finding about that server, and a strict run should say so.

Worth recording separately: none of the four attempts that succeeded used get_descendants. All four expanded by writing subClassOf* against ubergraph through the ordinary query tool. The tool the ablation exists to withhold is the one that broke the runs — which is also why the ablated arm appeared to beat the full toolkit, 0.67 against 0.33. That is not a finding about ontology tooling; it is the ablation accidentally routing around a broken upstream. The current numbers measure the outage far more than they measure the expansion. Re-run before drawing any conclusion from them, and see A caveat on the first sweep.

Also observed, and not yet fixed anywhere: get_join_strategy("nde", "oard-kg") on this server answers {"can_join": false, "common_identifiers": [], "strategy": "No shared identifier namespaces found between nde and oard-kg."}. That is wrong — both graphs key diseases on the same raw MONDO IRI, which is exactly what the answer key joins on — and agents called it in nine of the ten attempts.