benchmarks-single — one question, one knowledge graphThe 12-question floor corpus behind results-single/SUMMARY.md. No cross-graph join, no ablation, no unprompted ontology expansion.
This corpus is the baseline. Its job is to establish that the models can read a schema and write a correct query against one graph, so that failures in the cross-graph and ontology corpora can be attributed to joins and hierarchies rather than to basic competence. A ceiling measurement is only meaningful against a floor.
Each record is a prose #+ summary: — the prompt given to the agent — paired with a SPARQL answer key. The summary never names the knowledge graph; choosing it is part of the task, though the --kg-hint tags arm used in the recorded sweep does supply it. The summary states every limit and every tie-breaker, so the result set is fully determined by the prose. See the corpus format for the conventions all three corpora share.
One per knowledge graph, named for it.
| File | Asks for | Key rows |
|---|---|---|
biobricks-aopwiki_single.rq | adverse outcome pathways with TCDD as a stressor | 7 |
biomarkerkg_single.rq | biomarkers diagnostic for cirrhosis, with a specimen source | 40 |
digcfdekg_single.rq | top human genes for type 2 diabetes by association weight | 10 |
gene-expression-atlas-okn_single.rq | differential expression in study E-GEOD-54456 | 10 |
ncipidkg_single.rq | protein–protein relationships annotated as SUMOylation | 87 |
nde_single.rq | datasets per cardiovascular condition, subclasses included | 20 |
oard-kg_single.rq | top phenotypic features of Marfan syndrome | 10 |
pankgraph_single.rq | genes up-regulated in type 1 diabetes beta cells | 10 |
prokn_single.rq | proteins with the most Alzheimer-causing sequence variants | 10 |
rdkg_single.rq | mitochondrial rare diseases with the most phenotype terms | 10 |
spoke-genelab_single.rq | expression in the one balanced assay of spaceflight study OSD-244 | 10 |
spoke-okn_single.rq | New Mexico places by age-adjusted COPD prevalence | 10 |
Key shapes vary on purpose. Nine records are capped — eight at LIMIT 10, nde at 20 — and three are uncapped: biobricks-aopwiki (7 rows), biomarkerkg (40) and ncipidkg (87). A capped key measures ranking; an uncapped one measures membership, and an agent that quietly truncates is caught only by the second kind. Four records aggregate with COUNT DISTINCT (ncipidkg, nde, prokn, rdkg); the rest project stored values.
nde_single is the one exception, and it is deliberateEleven records read exactly one named graph. nde_single reads two: the NIAID dataset catalog and ubergraph, where it walks rdfs:subClassOf* from cardiovascular disease.
That is a second graph, so the "one graph per question" rule has an exception — but it is not a cross-graph join in the sense benchmarks-cross means, and it is not an unprompted expansion in the sense benchmarks-ontology means. The prompt says "counting cardiovascular disease together with every condition below it in the subclass hierarchy at any depth" — the hierarchy walk is stated outright, so nothing is being inferred. The record is here because a corpus of twelve one-graph questions with no ontology term at all would leave the harness's ontology path completely unexercised at the floor.
The consequence worth knowing: the server's ontology tooling is engaged by this corpus. get_descendants was called 12 / 8 / 7 times across the recorded Opus, Sonnet and Haiku sweeps. Do not read "single graph" as "no ontology".
For each of the twelve graphs there is one record here and one in benchmarks-cross/, anchored on the same graph. They are different questions — not the same question with a join bolted on — but the alignment is what lets a failure be attributed: rdkg_cross failing while rdkg_single passes says the agent can query rdkg and cannot join it. That attribution is the entire reason this corpus exists, so a new record here should keep the pairing intact.
Hard enough to be worth running, easy enough that a capable model passes. The recorded spread is 6/9 to 9/9 per record, against 2/9 to 9/9 in the cross-graph corpus. That compression is the floor being a floor: if a record here separates the models sharply, it is probably testing something other than plain SPARQL competence and belongs in another corpus.
Some records must be answerable by everyone. Four are 9/9 — gene-expression-atlas-okn, ncipidkg, oard-kg and spoke-okn. They are worth keeping precisely because they are easy: a corpus with no question that everybody answers cannot distinguish "this model is weak" from "this record is broken."
The prose still has to be airtight. Being a floor is not license to be vague. ncipidkg_single spends three lines of its summary on one grain question — a few protein pairs are described by more than one statement record, carrying the same evidence count — because "list each pair once with that count, neither repeating the pair nor adding the counts together" is the difference between two defensible answers. biomarkerkg_single spends a clause distinguishing diagnostic for from indicating risk, prognosis or ongoing status, the same distinction its cross-graph counterpart turns into a trap. And spoke-genelab_single is the longest summary in the corpus, because "the assay where the two groups are otherwise identical" has to be defined — setting aside the spaceflight and ground-control labels themselves, neither group may carry any factor the other does not — before the record has one answer instead of several.
No cross-graph join, so identifier crosswalks are never exercised and the get_join_strategy defect the other two corpora surface cannot appear here. No unprompted ontology expansion — the one hierarchy walk is spelled out in the prompt. No ablation, so the value of the toolkit is not measured.
Those omissions are the point. This corpus answers one question — can the model query a single graph correctly — and every richer claim belongs to the corpus that adds a variable.
Full results are in results-single/. Per record, out of 3 attempts each for Opus 5 / Sonnet 5 / Haiku 4.5, counting an attempt only when the rows and their order are right:
| Record | Opus | Sonnet | Haiku | Total |
|---|---|---|---|---|
gene-expression-atlas-okn | 3 | 3 | 3 | 9/9 |
ncipidkg | 3 | 3 | 3 | 9/9 |
oard-kg | 3 | 3 | 3 | 9/9 |
spoke-okn | 3 | 3 | 3 | 9/9 |
digcfdekg | 3 | 3 | 2 | 8/9 |
spoke-genelab | 3 | 3 | 2 | 8/9 |
pankgraph | 3 | 2 | 2 | 7/9 |
biobricks-aopwiki | 3 | 3 | 0 | 6/9 |
biomarkerkg | 3 | 1 | 2 | 6/9 |
nde | 3 | 3 | 0 | 6/9 |
prokn | 3 | 3 | 0 | 6/9 |
rdkg | 3 | 1 | 2 | 6/9 |
Opus answers 36 of 36. That is the number the other two studies are read against: every Opus failure recorded elsewhere is a failure at joining, at expanding a hierarchy, or at species discipline — not at reading a schema and writing a query.
Haiku misses 14 of 36 with nothing in the way, and three records defeat it entirely. That reframes its results elsewhere: its 7/36 on the ontology corpus cannot be read as an ontology-specific weakness when it is already missing more than a third of the floor.
No attempt in this corpus hit the agent deadline, so unlike the ontology sweep every timing here is a clean measurement rather than a truncated one.
okn-benchmark reference --corpus benchmarks-single
okn-benchmark run --corpus benchmarks-single --tools all-tools --kg-hint tags \
--profile profiles/mcp-proto-okn.local.toml --model claude-opus-5 --effort high \
--repeat 3 --agent-timeout 900 --out runs-opus-singleOne arm only. Withholding tools measures what the tooling is worth, which is the cross-graph study's question; adding that axis to a floor measurement would only make the floor harder to read.