benchmarks-cross — questions no single knowledge graph can answer

The 12-question cross-graph corpus behind results-cross/SUMMARY.md. Every record joins two knowledge graphs, and neither graph alone holds the answer.

What it tests: whether an agent can pick two graphs out of twenty-seven, find the identifier they have in common, and join on it correctly. That is three separable skills, and a record can fail at any of them — the corpus does not try to isolate which, because the single-graph corpus already establishes that the models can query one graph on its own. A failure here that is not a failure there is a failure at the join.

Each record is a prose #+ summary: — the prompt given to the agent — paired with a SPARQL answer key. The summary never names either graph; choosing them is half the task. It states every limit and every tie-breaker, so the result set is fully determined by the prose. See the corpus format for the conventions all three corpora share.

The twelve records

Named for the graph the key anchors on — the smaller, more selective side, which every key touches first. The partner is the second tag.

↺ marks the five records whose join needs an identifier rewrite. The other seven join on an IRI both graphs mint identically. That split is deliberate and is the corpus's main axis of difficulty: a direct join needs the agent to notice that two graphs share a key, while a rewrite needs it to notice that they nearly share one.

The rewrites, in full — each is one BIND in the key:

Note that biobricks-aopwiki and digcfdekg rewrite the same pair of namespaces in opposite directions. Neither direction is canonical; which one a record uses is decided by which graph it anchors on.

All twelve graphs are covered, but not evenly. Twelve pairs over twelve graphs cannot be a perfect matching once each graph anchors exactly one record: prokn and spoke-okn appear three times, ncipidkg and nde once, the rest twice.

Key sizes. Eleven records return exactly 10 rows under an explicit LIMIT 10. nde_cross has no LIMIT — it asks for all diseases meeting two conditions, and there happen to be 8. A capped key measures ranking; an uncapped one measures membership, and the corpus wanted at least one of the latter.

Aligned with the single-graph corpus

For each of the twelve graphs there is one record here and one in benchmarks-single/, anchored on the same graph. They are different questions — not the same question with a join bolted on — but the alignment is what lets a failure be attributed: rdkg_cross failing while rdkg_single passes says the agent can query rdkg and cannot join it. Read the two corpora together for exactly this reason.

What actually makes a record hard

Twelve rounds of "add a join" would have produced twelve variations on one difficulty. These are the distinct traps the corpus ended up encoding instead, each discovered by watching a record fail:

A polymorphic predicate. biomarkerkg_cross is the corpus's hardest question by a wide margin — 9 of 27 attempts across all models and arms. The property linking an adverse-outcome-pathway event to its protein target, edam:data_1025, is asserted on both aop:KeyEvent and aop:KeyEventRelationship, and only the first is a key event. An agent that omits the type restriction gets 43 links where it should get 18. The full diagnosis is in results-cross/QUESTIONS.md.

A crosswalk predicate that means something else. ncipidkg_cross joins UniProt to UniProt with no rewrite at all, because both graphs already mint purl.uniprot.org. The trap is the tempting hop: in ncipidkg, owl:sameAs maps a canonical accession to its secondary and obsolete accessions (P04637 → Q15086, Q15087, …), so an agent that follows it to "normalize" the identifier lands on merged-away entries and silently answers a different question.

A counted unit that is not the obvious one. spoke-genelab_cross counts assays in a graph that asserts the same attribute on studies as well as assays — 11,313 assay subjects against 2,472 study subjects. Dropping the type restriction still returns every correct member and gets the order wrong, scoring F1 0.97 rather than visibly breaking. The prose now says "count only its assays" outright, because a trap that produces a near-miss is a bad trap: it is indistinguishable from noise.

A filter that changes the answer without touching the ranking key. Also spoke-genelab_cross. Restricting to flight-flagged assays cuts 7,106 assays to 2,022 and drops lung and embryo out of the join entirely — even though the ordering key comes from the other graph and no filter touches it. Inner joins propagate filters across the join in a way that is easy to reason past.

A cross-graph count that cannot be stated unambiguously. gene-expression-atlas-okn_cross deliberately carries none. Two earlier versions ranked by "count of things in the other graph" and both failed blind runs on what the counted unit was — association records or assays. The record now ranks by the anchor study's own log2 fold change and lets the partner graph contribute a categorical label instead of a number.

Cautions for anyone adding a record

Verify the record before building it. Measure the density and the discriminating power of the join key first. A graph's advertised identifier_namespaces are frequently crosswalk annotations rather than primary identifiers, and one candidate graph in this project advertised HPO while carrying it on about 1% of its traits — not enough to key an answer against.

Anchor on the selective side. Every key here binds the smaller graph first, and several say so in a comment. Anchoring on the large side turns a fast query into a federation-wide scan, and the 900s agent deadline is real.

State every limit in the prose. The result set has to be fully determined by the summary alone. spoke-genelab_cross was reworded twice after runs showed its prose admitted a second honest reading — once for the control assays, once for studies-versus-assays. When an agent's wrong answer is defensible from the prompt, the record is wrong, not the agent.

Prefer a trap that breaks the answer to one that perturbs it. See the assay/study note above.

Do not trust the server's own join recipe. For the ncipidkg × prokn pair the federation's cataloged recipe prescribes an identifiers.org rewrite that returns 0 rows today. The key here joins directly and says so in a comment.

How the corpus behaved

Full results are in results-cross/. Per record, in the fully-equipped arm (all tools, graph named), out of 3 attempts each for Opus 5 / Sonnet 5 / Haiku 4.5:

The 2/9 to 9/9 spread is what a corpus needs: one record everybody answers, one record almost nobody does, and ten in between. Compare the single-graph corpus, where the same twelve graphs produce a 6/9-to-9/9 spread — that compression is the floor being a floor.

Running it

okn-benchmark reference --corpus benchmarks-cross
okn-benchmark run --corpus benchmarks-cross --tools all-tools --kg-hint tags \
  --profile profiles/mcp-proto-okn.local.toml --model claude-opus-5 --effort high \
  --repeat 3 --agent-timeout 900 --out runs-opus-cross

The recorded sweep adds two ablated arms — --kg-hint none, and six tools withheld at the proxy. results-cross/SUMMARY.md §12 has the full command set.