mcp-proto-okn-benchmark

A benchmark harness for MCP servers over the Proto-OKN knowledge graphs. It measures how well an agent turns a prose question into a correct SPARQL answer, and how much of that success comes from the server's tooling.

It is server-agnostic by design: nothing here is specific to one server, and a second one is added by writing a profile. The examples throughout use mcp-proto-okn, running locally over stdio or remotely over streamable HTTP.

Results so far — published at sbl-sdsc.github.io/mcp-proto-okn-benchmark, where the figures render.

Three corpora ship in this repo, and each has its own completed three-model study — Opus 5, Sonnet 5 and Haiku 4.5, three attempts per record, 540 recorded episodes in all. They are meant to be read in this order, because each one is the control for the next.

The floor comes first because it is what makes the other two readable. Opus answers 36 of 36 single-graph questions, so every Opus failure recorded in the other two studies is a failure at joining or at expanding a hierarchy — not at reading a schema and writing a query. Haiku, by contrast, already misses 14 of 36 at the floor, which is why its weak ontology score cannot be read as an ontology-specific weakness.

Only the cross-graph study has an ablation axis. It runs three tool conditions — the full toolkit with the graph named, the full toolkit with the graph withheld, and six tools withheld at the proxy — which is what lets it price the tooling. The other two run one condition each: adding that axis to a floor measurement would only make the floor harder to read.

Each study directory holds a SUMMARY.md, the figures behind it, and the scripts that rebuild both from the recorded episodes. Each also holds a QUESTIONS.md reading the same episodes record by record — single, cross, ontology — which is where a question that defeats every model, or one that separates them, is named.

Every corpus directory has a README explaining what its records test, what makes one hard, and how to add another. The runs behind every number are in runs-{opus,sonnet,haiku}-{single,cross,ontology}/, one directory per model per corpus, each with a generated README of its own.

Every README here also exists as self-contained HTML beside it — the seven prose ones and the nine generated per-root reports — because GitHub Pages serves a .md file as plain text rather than rendering it. Rebuild after editing a README, or after any run, rescore or report that rewrites a runs root's own:

uv run python build_readme_html.py            # rewrite all sixteen
uv run python build_readme_html.py --check    # fail if any is out of date; CI runs this

Two further scripts serve the manuscript rather than the repository. build_paper_tables.py reads the recorded episodes and prints the paper's tables as markdown, so a number in the manuscript cannot drift from the study it came from. build_paper_doc.py converts a markdown manuscript to HTML and RTF, optionally one file per table, for submission systems and word processors that will not take markdown. The manuscript itself is not kept here.

uv run python build_paper_tables.py                       # tables, to stdout
uv run python build_paper_doc.py --tables paper.md        # paper.html, paper.rtf, paper-tableN.rtf

Quick start

uv sync                                    # no ANTHROPIC_API_KEY needed — see Authentication
export CORPUS=benchmarks-cross                   # the corpus ships in this repo

uv run okn-benchmark reference --corpus "$CORPUS"          # build ground truth (no model)
uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools all-tools --out runs-demo \
    --profile profiles/mcp-proto-okn.remote.toml \
    --kg-hint tags --model claude-opus-5 --effort high --repeat 3 --limit 2
uv run okn-benchmark report runs-demo --markdown runs-demo/README.md

Every subcommand takes --help. The rest of this file explains the inputs, the flags and what lands on disk.

The three inputs

The corpus

A corpus is a directory of .rq files. Each pairs a prose #+ summary: — the prompt given to the agent — with a SPARQL query, the answer key. Three ship here — benchmarks-single/, benchmarks-cross/ and benchmarks-ontology/, twelve records each — and every one of them has a README describing its records. This is one from benchmarks-cross/:

#+ summary: Among places in New Mexico with a recorded total population of at least 5,000,
#+          return the 10 with the highest age-adjusted prevalence of chronic obstructive
#+          pulmonary disease, reported as a percentage. ...
#+ tags:
#+   - spoke-okn

PREFIX ...
SELECT DISTINCT ?placeName ?stateDesc ?prevalence ?population ?place
WHERE { GRAPH <https://purl.org/okn/frink/kg/spoke-okn> { ... } }
ORDER BY DESC(?prevalence) ASC(?placeName) ASC(?place)
LIMIT 10

The summary never names the knowledge graph — choosing it is half the task. It states every limit and every tie-breaker, so the result set is fully determined by the prose.

Tiers are discovered, not hardcoded. A file's tier is whatever follows the last underscore in its name: spoke-okn_hard.rq → hard. Add a spoke-okn_medium.rq and a medium tier exists, with no code change. --tier X filters; omitting it runs everything.

--corpus accepts either a local directory or a git repository:

--corpus benchmarks-cross                                               # a local directory
--corpus 'git+https://github.com/user/repo'                       # default branch
--corpus 'git+https://github.com/user/repo@v2#benchmarks-cross/set1'    # ref and subdirectory

A git corpus is shallow-cloned into a local cache and reused. Its resolved commit is recorded with the run, so a result stays tied to the exact questions that produced it.

--limit N caps how many questions run — useful for a smoke test before committing to a full sweep.

--exclude PATTERN keeps a question out. The pattern is a glob over the record id, so one oard-kg_* covers that question in every tier; the flag is repeatable, and applies before --limit so the cap counts what will actually run.

Check it against the corpus before spending a sweep on it — corpus list applies the same selection run would, and prints how much of the corpus survives:

uv run okn-benchmark corpus list --corpus "$CORPUS"                        # what ids exist
uv run okn-benchmark corpus list --corpus "$CORPUS" --exclude 'oard-kg_*'  # "11 of 12 records"
uv run okn-benchmark run --corpus "$CORPUS" --exclude 'oard-kg_*' ...

A pattern matching nothing is an error, not a no-op. Its whole purpose is to keep a question out of a run, and a typo would keep nothing out while looking exactly like success — you would find the question in the results afterwards. The same error is how you learn a pattern has gone stale because the corpus moved on: a pattern that matched yesterday and matches nothing today says so, rather than silently excluding nothing.

Matching is on the id rather than the tag or the tier because ids are the only stable handle: tiers are discovered from filenames and are never named in this package, so a flag that matched on tier would be the one place contradicting that.

The flag is shared by corpus list, reference, selfcheck and run, so a question can be kept out of ground truth as well as out of the sweep. run records what it dropped in conditions/<name>.json, and report <root> names it beside the counts — a corpus of thirteen with one question withheld is a different claim from a corpus of twelve. The generated markdown leaves it out, so the shareable summary stays about the results; the provenance is in the JSON and in the terminal report. What was excluded is also part of a condition's identity, so a 12-question run cannot silently blend into a 13-question folder.

What the agent is actually asked

The prompt is the summary, optionally a KG hint, then a fixed suffix — identical in every arm, so no condition is advantaged by its wording:

<the #+ summary: text>

Use this knowledge graph: spoke-okn.        ← only when --kg-hint tags

Save the results as a csv file, no prose!
Call the submit_csv tool with the complete result table. Use a header row of
column names. If nothing satisfies the question, submit the header row on its
own — an empty result is an answer, and is not the same as not answering. Write
IRIs in full — the complete IRI, not a prefixed or abbreviated form.

The assembled text is written to prompt.txt in every attempt directory, so what the agent received is auditable rather than reconstructed from the flags.

The empty-result sentence is there because "the answer is the empty set" is a finding and "I could not do this" is a failure, and an agent needs a way to say the first. Without it, "the complete result table" reads as do not call the tool when there is no table — so an agent that correctly finds nothing submits nothing, and lands in the same bucket as one that timed out. The harness distinguishes them either way (a header-only submission records submitted: true with zero rows; a silent one records submitted: false), but only if the agent knows the move is available. It sits in the shared suffix rather than in the records that can actually return nothing: said to one question it would hint that empty is plausible there, said to every question it singles out none of them.

The profile

A profile is a small TOML file naming only the transport. Nothing about tools is configured — the proxy discovers the tool list from the server itself, so a server that gains or loses a tool needs no profile edit.

Remote, over streamable HTTP:

name = "mcp-proto-okn"

[upstream]
transport = "http"
url = "https://apps.okn.us/okn-mcp/mcp"
# headers = { Authorization = "Bearer ..." }   # optional; omitted from the record

Local, over stdio:

name = "mcp-proto-okn"

[upstream]
transport = "stdio"
command = "uv"
args = ["run", "--directory", "~/GitRepositories/mcp-proto-okn", "mcp-proto-okn-unified"]
# cwd = "..."
# env = { KEY = "value" }

name is what appears in condition folder names and reports. Profiles live in profiles/, each server as a .local.toml and a .remote.toml — mcp-proto-okn is the one used throughout this file.

Any headers are omitted from conditions/<name>.json — they may carry credentials, and that file is meant to be shareable.

--profile is required for --tools all-tools and ignored for --tools sparql-only, which has no upstream.

Check a profile before spending model time:

uv run okn-benchmark probe --profile profiles/mcp-proto-okn.remote.toml

It connects, prints the tool count and instruction length, shows what the agent would see at each level, and prints the sparql-only preamble digest — which must match across profiles, since that is what makes sparql-only a shared control.

Choosing the model and effort

--model claude-opus-5      # exact model id — preferred
--model opus               # alias; the default, but it moves over time
--effort high              # low | medium | high | xhigh | max

Both are passed through to claude -p and recorded with the condition.

Prefer an exact model id. opus is an alias that resolves to whatever is current, so a result attributed to it is not reproducible later. The harness records model_resolved from the first episode's own report either way, and the generated report shows the exact id with the alias beside it — but naming the id up front removes the ambiguity entirely.

--effort has no default of its own: omit it and the run takes the CLI's own default, which is what the reports and figures call default. Set it explicitly whenever you intend to compare two runs, or the comparison silently spans two effort levels.

Both fields are part of a condition's identity, so changing either makes run refuse to overwrite an existing condition folder (see Repeating and comparing).

What gets measured

Two tool levels, so the contribution of the server's tooling can be read off directly:

all-tools − sparql-only is what the server's tools and instructions add over a bare ability to run SPARQL. sparql-only is server-independent, so it is run once per corpus and tier and reused as the shared control for every server, rather than re-run for each. That "once" is about the server axis only: --repeat applies to it exactly as to any other condition.

A second axis — whether the prompt names the target KG (--kg-hint none|tags) — applies to all-tools only, where tags − none is what naming the graph saves. Together those give three conditions, not a 2×2:

The control is always given the graph. It has to be handed the list of valid shortnames simply to address the endpoint, and given that list it recognizes the right graph rather than choosing one — so withholding the name there measures name recognition, not graph selection: across 72 recorded episodes the floor's first query hit the target graph 59 times, with no exploratory calls at all. run refuses that combination.

Withholding individual tools

all-tools − sparql-only is an aggregate. It cannot say which tools carried the gain, and routers and crosswalk helpers are the expensive ones to build. --without-tool NAME (repeatable) withholds named tools from an otherwise-full all-tools arm, so the drop is what those tools were worth:

okn-benchmark run --corpus benchmarks-cross --tier cross --tools all-tools \
  --profile profiles/mcp-proto-okn.local.toml --kg-hint tags \
  --without-tool get_schema --without-tool route_query --out runs-opus-cross

Crossed with the KG-hint axis this gives a four-cell study that needs no floor at all — and that is the point, because the floor cannot sit on both sides of the KG axis:

all tools − ablated is what the withheld tools add; tags − none is what naming the graph saves; and because the axes cross you can also see whether a tool's value depends on the agent already knowing the graph — the question for a router like route_query.

Three things make the measurement trustworthy:

• Withholding is enforced on both halves of the protocol. Hiding a tool from tools/list is not enough: the proxy's tools arrive deferred, so the agent finds them through ToolSearch and a remembered name would still be forwarded. A call to a withheld tool is refused at the proxy and never reaches the server.
• The names are checked against the server before anything runs. A typo would withhold nothing, score like the full toolkit, and report with confidence that the tool was worth zero. Preview an arm with probe --without-tool NAME, which costs no tokens.
• Reaching for a withheld tool is recorded. The refusal is logged, so the report can say the agent had a use in mind for a tool it did not have — which no score can show.

submit_csv can never be withheld: it is injected after the filter, because an episode with nowhere to put its answer measures nothing.

Alongside the score, every run records which tools were actually called and in what order, how many queries ran (split succeeded/errored), and elapsed time decomposed into server time and agent time. A tool that is never called contributes nothing regardless of the score.

Closing a runs root

Results are generated as well as recorded: README.md and summary.csv are rewritten by run, report and rescore, and rescore rewrites every result.json in place. Once a set of results is published that is a liability — one mistyped --out, or one change to the report format, silently rewrites them. freeze closes a root:

okn-benchmark freeze runs-opus-cross --reason "Published; new runs go elsewhere."

run and rescore then refuse it by name, and report reads it without regenerating anything. Unfreezing means deleting the .frozen marker by hand.

How it works

  runner            `claude -p`, authenticated by your existing Claude Code login
     │  stdio, --strict-mcp-config
     ▼
  proxy             re-exposes a filtered tool set · strips or forwards the server's
     │              instructions · injects submit_csv · times every call
     ▼
  server under test   local: stdio subprocess   ·   remote: streamable HTTP

The proxy is the ablation surface, for the level and for --without-tool alike — it filters tools/list and refuses tools/call, so a withheld tool is gone rather than merely hidden. It is also why local and remote are interchangeable: the transport choice lives entirely in its upstream connection.

The commands

The four free commands catch nearly everything that can go wrong. Run them first.

1. Inspect and prepare

uv run okn-benchmark corpus list --corpus "$CORPUS"     # what is in it, which tiers
uv run okn-benchmark reference --corpus "$CORPUS"       # build ground truth
uv run okn-benchmark selfcheck --corpus "$CORPUS"       # scoring wired correctly?
uv run okn-benchmark probe --profile profiles/mcp-proto-okn.remote.toml

reference is a prerequisite for run — without cached ground truth there is nothing to score against, and run will tell you so and stop. Re-run it after editing any answer key.

2. Run the conditions

One invocation runs one condition — a tool level crossed with a KG-hint mode — over the whole corpus. --out is the runs root, shared by every condition; the condition's own folder name is derived from the flags, so the label and the settings cannot disagree.

There is no matrix subcommand: the full matrix for one server is three invocations, and running fewer leaves cells missing from the report rather than failing loudly.

One root per tier. A condition's folder name carries the tool level, the server and the KG hint — but not the tier. Two tiers in one root would therefore write to the same conditions/<name>.json; the collision guard catches it and stops the run, and the fix is a separate root. Name it after what it holds:

# The control: a neutral SPARQL tool, no server, no instructions. KG named only —
# with it unnamed the floor does not select the graph, it reads it off the
# shortname list, so `run` refuses that combination.
uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools sparql-only \
    --kg-hint tags --out runs-demo \
    --model claude-opus-5 --effort high --repeat 3

# The server under test, both KG-hint modes.
for hint in none tags; do
  uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools all-tools \
      --kg-hint "$hint" --out runs-demo \
      --profile profiles/mcp-proto-okn.remote.toml \
      --model claude-opus-5 --effort high --repeat 3
done

Another tier is the same invocations with --tier X --out runs-X. Each root then gets its own README.md and summary.csv, and cross-tier comparison is done by reading the two — deltas are only ever computed within a corpus and tier, never across.

The study in results-cross/SUMMARY.md drops the floor and uses --without-tool for its third arm instead; see Withholding individual tools for why the floor cannot sit on both sides of the KG-hint axis.

sparql-only is server-independent, so run it once per root and reuse it as the control for every server — its condition folder carries no server name, which is what makes the reuse possible. Point a second server at the same --out and the report grows a column against the same control. This is the one axis it is not repeated along; --repeat 3 still attempts each of its questions three times, like any other condition.

Questions run strictly one at a time, and so do attempts. Timings are meaningless under contention, which is why nothing here is parallel.

Resuming an interrupted run

A condition is only recorded once it finishes: conditions/<name>.json is written at the end, so a sweep killed halfway leaves attempt directories on disk and no condition. --resume picks it up from there.

uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools all-tools \
    --kg-hint tags --out runs-demo --profile profiles/mcp-proto-okn.local.toml \
    --model claude-opus-5 --effort high --repeat 3 --resume

Each attempt already recorded under --out is read back off disk instead of being run again; the rest run normally. Reused attempts are marked [resumed] in the output, so a resumed run never reads as a fresh one.

Reuse is conditional. An attempt is only accepted when its recorded command.txt and prompt.txt match what the invocation would issue, byte for byte — otherwise it is re-run. That is the guard that matters when swapping a profile: pointing --profile at a local checkout instead of the deployment leaves the condition name, the model and the effort all unchanged, so nothing else on disk would notice half the attempts belonging to a different server.

What makes an attempt resumable is episode.json, written last, holding what the runner itself observed — wall time, exit code, resolved model. An attempt killed mid-episode has no such file and is simply run again. A timeout, by contrast, is a recorded outcome and is reused rather than paid for twice.

Attempts written before episode.json existed are still honoured, recovered from the agent host's own agent.json and stamped "source": "agent.json" — its duration_ms is the agent's clock and excludes process startup, so a wall time recovered that way is marked rather than passed off as measured.

3. Report

uv run okn-benchmark report runs-opus-cross                # terminal tables
uv run okn-benchmark report runs-opus-cross --markdown runs-opus-cross/README.md
uv run okn-benchmark report runs-opus-cross --markdown -   # markdown to stdout

Repeating and comparing

--repeat N attempts each question N times. One episode cannot tell "cannot do this" from "does this sometimes"; repeats separate them, and the report gains always exact, ever exact and a flaky count. Under repeats every per-question figure is a per-attempt average.

A condition's identity is level, kg_hint, server, model, effort, tier and repeat. Two runs differing in any of them would derive the same folder name and silently overwrite, so run refuses and names the first field that disagrees:

'mcp-proto-okn-all-tools__kg-named' already holds a run with different settings:
  effort: 'high' -> 'low'

Use --label NAME to keep them apart, or --force to replace.

• --label NAME — keep both side by side, e.g. when sweeping effort levels
• --force — replace the existing condition

Authentication and cost

No ANTHROPIC_API_KEY is needed. The runner drives the claude binary already on your PATH, which uses your existing Claude Code login, so runs consume Claude Code rate limits rather than API credit.

Each episode is one claude -p invocation. A sweep is conditions × questions × repeat episodes, run serially — four conditions over 26 questions at --repeat 3 is 312 episodes, so start with --limit or a small corpus.

Two guards on a long run: --agent-timeout (default 900s per episode) and --max-turns.

When the server under test fails quietly

A tool can report success and put the failure in the payload. The remote get_descendants does exactly this when Ubergraph rate-limits it — isError=false, descendant_count: 0, and an HTTP Error 429 in the body — and to every counter that reads as a successful call returning an empty set. An episode derailed by one is then scored as an agent that chose not to expand.

Two things guard against that:

• Every forwarded call carries a read timeout — the proxy's --timeout, 120s by default. Unbounded is the MCP client's default and it is the wrong one here: a server that accepts tools/call and then loses the response leaves the session blocked with no way out, so the episode spends the rest of its budget waiting and is filed as an agent timeout. Observed as a 900s episode whose call log stops mid-run after 35s of server time. Bounded, the same thing becomes an ordinary failed tool call the agent can act on.
• A failure hidden in a successful reply is recorded. Each call in calls.json carries a soft_error, and metrics carries an upstream_soft_errors tally per tool. ok stays what the protocol said, because "the tool failed" and "the tool said it succeeded and failed anyway" are different facts about a server.
• A session that dies quietly is caught too. The holder task below handles a connection that raises on its way out, but a session can also just stop answering: the transport closes, every call returns "Session terminated", and nothing raises. Three consecutive failed forwarded calls declare the session dead, which needs no guess about which exception a dead transport throws. Any success resets the count, so an intermittently-failing server is never cut off. Observed before this existed: one query hit its read timeout and the next 73 calls all failed the same way while the agent polled SELECT ?s WHERE { ?s ?p ?o } LIMIT 1 waiting for it to recover. With the breaker the same episode ends itself in 190s instead of burning a 480s budget.
• The proxy outlives its upstream. A session can die mid-episode — the same 429 has taken the whole streamable-HTTP connection down with it. That used to end the proxy process, and with it submit_csv, which is served locally and needs no upstream at all. The connection is now held beside the stdio server: losing it marks the session, is recorded as lost_upstream in metrics (distinct from startup_error — an episode that never had tools is not one that lost them part-way), and every later call fails with a reply saying the connection will not come back and that the answer channel is still open. Without that last part an agent reads the error as transient and retries into a dead socket until its budget runs out.

--strict-upstream-errors goes one step further and relabels such a reply as an error before the agent sees it. The payload is forwarded verbatim either way; only the error flag differs. It is off by default, and the default is the honest one — a server that fails softly is a finding about that server, and hiding it behind the proxy would flatter the thing under test. It is part of a condition's identity, so a strict arm and a plain one cannot blend into one folder.

Re-scoring instead of re-running

uv run okn-benchmark rescore runs-opus-cross

Everything the scorer needs is already on disk — the submitted answer.csv and the ground_truth beside it — so rescore re-judges without a model, a server or an endpoint, and rewrites result.json, conditions/*.json, both summary CSVs and README.md. It prints every score that moved.

Use it whenever scoring changes. Re-running instead spends model time to correct arithmetic and produces different episodes, so the thing being compared moves underneath you. Re-scoring holds the episodes fixed and changes only the judgment.

It does not re-run answer keys. If the corpus itself changed, that calls for a new run.

When an episode never reaches the tools

If the proxy cannot reach the server under test it dies at startup, the agent host registers no tools, and the agent correctly reports that it cannot do the task. That is not a wrong answer, and counting it as one attributes a plumbing failure to the server.

Such an episode is marked blocked and excluded from every rate — but never silently. The run output, the per-question row and the summary CSVs all carry the count and the recorded reason:

| mcp-proto-okn, all tools · KG named | 2/2 | ✓ | 1.00 | 13.5 | ... | 1 of 3 never reached the tools |

Blocking requires positive evidence: an episode that submitted an answer is never blocked, and an episode with no metrics at all stays a miss. Guessing from absent evidence would drop real failures out of the denominator, which is the one direction that inflates a score.

If a whole question is blocked throughout, it reports as measured nothing rather than as a pass or a failure.

Scoring

Answers are compared by denotation, not by SELECT clause: column names and column order are ignored, so a correct answer may name its variables freely.

Where the answer uses the reference's column names, values are matched column by column. Only where the names differ does a row fall back to the sorted tuple of its values. Sorting is what makes names irrelevant, but it also makes same-typed columns interchangeable — a row reversing source and target sorts identically to the original — and three corpus questions pair columns that way, so sorting is the fallback rather than the rule.

Two further adjustments make that fair across servers:

• IRI normalization. Servers serialize IRIs differently — MONDO_0000745, obo/MONDO_0000745, and http://purl.obolibrary.org/obo/MONDO_0000745 all occur. IRI-ish cells are reduced to their local name, guarded by a per-question check that the reduction is injective on the reference; if two distinct reference IRIs would collide, that question falls back to strict comparison.
• Numeric tolerance. Computed decimals (AVG, division) print differently across engines, so non-integral numbers are rounded to a fixed number of significant figures rather than compared as strings — and, because rounding converges only when the target is coarser than both sides, a candidate decimal is additionally snapped onto the reference value it is printing when one lies within a relative tolerance of 5e-4. Without that, an agent rendering 0.45356 for a reference 0.453562 scores zero for a right answer, and no choice of precision fixes it — rounding an already-5-figure value to 6 figures is a no-op. Snapping is withdrawn for the whole question if the reference's own decimals sit closer together than the tolerance, mirroring the IRI injectivity guard: where the reference cannot separate two values, the scorer must not pretend it can.

Both strict and normalized scores are reported. The gap between them measures how far the servers' serializations diverge, which is worth seeing rather than hiding.

Order matters too: the summaries state a total ORDER BY with named tie-breakers, so an order-sensitive exact match is reported alongside the order-insensitive one.

Where the results go

Results are organized by question, so every condition's answer to one question sits beside the expected rows rather than in four sibling directories:

runs-opus-cross/
  README.md                          the generated summary, with a table per question
  summary.csv                        one row per question × condition × attempt
  conditions/
    mcp-proto-okn-all-tools__kg-named.json      settings + aggregate, recorded once
    mcp-proto-okn-all-tools__kg-not-named.json

  spoke-okn_cross_results/
    ground_truth.csv                 the expected rows, in the shape the agent submits
    ground_truth.json                endpoint, engine, the answer key and its fingerprint
    summary.csv                      this question across every condition and attempt
    mcp-proto-okn-all-tools__kg-named/
      result.json
      attempt-1/
        prompt.txt  answer.csv  calls.json  agent.json  command.txt  episode.json  cwd/
    mcp-proto-okn-all-tools__kg-not-named/
      ...

ground_truth.csv is written in the answer key's SELECT order, so it diffs directly against any attempt's answer.csv — a correct answer differs only in the header row.

summary.csv carries one row per attempt, not per condition — question, tier, condition, server, tools, kg_hint, attempt, attempts_total, blocked, exact, ordered, f1, queries, tools_called, wall_s, server_s, notes. A question answered 2 of 3 times shows as three rows; averaging it into one is how flakiness hides. Filter on blocked before averaging anything.

The f1 column is partial credit for a near-miss, computed per attempt over the answer's rows as a multiset: precision = matching rows ÷ rows the answer returned, recall = matching rows ÷ rows the key returned, F1 = 2PR/(P+R). A row matches only if it agrees cell for cell, after the same adjustments exactness uses — IRIs by local name where unambiguous, decimals at --precision significant digits within numeric_rel_tol. Column names are ignored unless both sides use the same set, in which case cells are matched column by column. Both sides empty is 1.0; no overlap is 0.0. The f1 reported here is the IRI-normalized tier. Any mean F1 in the reports averages a question's attempts first and then the questions, so each question weighs the same.

Inside an attempt directory:

The scratch cwd/ is deliberately empty: Claude Code's auto-memory and project CLAUDE.md are keyed by working directory, so running there means neither reaches the agent and every episode starts from the same state.

Two other things are kept out, neither of which --setting-sources "" covers:

• Skills, via --disable-slash-commands. Skills are not settings, so a personal or plugin skill would reach the agent, and the CLI's own bundled ones are advertised in context on any machine. --bare is no substitute — it states outright that skills still resolve.
• Delegation, via --disallowed-tools: subagents, workflows, background tasks, cron, messages to other sessions. An episode that answers by spawning something else is not the episode being scored, whatever tools the delegate turns out to have.

ToolSearch is deliberately kept. The proxy's own tools arrive deferred, so it is how the agent reaches run_sparql at all.

That list is a blocklist, and blocklists rot — it is only as complete as the tool surface on the day it was written. After upgrading claude, run one episode with these exact flags and ask it to name every tool and skill it has. Anything but ToolSearch and mcp__okn__* belongs in DISALLOWED.

Reproducibility

Every run writes conditions/<name>.json recording the corpus and its git ref, the profile, tool level, KG-hint mode, model, effort, claude --version, the reference endpoint and engine, the sparql-only preamble verbatim, the scoring precision and numeric tolerance, and — for a remote server — the build hash it was serving. A number is attributable to a specific deployment and a specific agent build, not to "the server" in the abstract.

Ground truth is fingerprinted per question: editing an answer key invalidates that question's cached rows and nothing else, so a subsetted run (--tier, --limit) is not falsely reported as stale.

Exclusions are recorded rather than merely absent. run --exclude writes them to meta.excluded, and report <root> names them beside the counts:

"excluded": [{"id": "oard-kg_cross", "reason": "excluded by --exclude oard-kg_*"}]

To drop a question from results that already exist, without re-running: remove its entry from each conditions/<name>.json, add it to that file's meta.excluded by hand in the same shape, and rescore to recompute the aggregates and both CSVs. Only the ids take part in a condition's identity, so a hand-written reason and a --exclude one describe the same run.

command.txt embeds absolute log paths in its --mcp-config, so it re-runs in place.

Questions run strictly one at a time. Timings are meaningless if anything else is contending for the endpoint.

Development

uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest

CI runs three jobs: lint (ruff check, ruff format --check, then mypy over src/okn_benchmark), tests across the supported Python versions, and a guard that no tier name may appear in code. Run both ruff commands locally — a formatting-only difference turns CI red on its own.

That last job is why tiers are discovered from filenames. If "simple" or "hard" were ever written into the source, a corpus introducing a new tier would silently stop working, so it fails the build rather than waiting to be noticed.

Citation

If you use this benchmark, please cite the paper describing the server it was built to measure:

Rose, P. W., Good, B. M., Saravia-Butler, A. M., Nelson, C. A., Balhoff, J. P., Kebede, Y., Whetzel, P. L., Bizon, C., Su, A. I., & Baranzini, S. E. (2026). mcp-proto-okn: Natural-language access to open scientific knowledge graphs through the Model Context Protocol. arXiv:2605.30283. <https://arxiv.org/abs/2605.30283>

@misc{rose2026mcpprotookn,
  title         = {mcp-proto-okn: Natural-language access to open scientific knowledge graphs through the Model Context Protocol},
  author        = {Rose, Peter W. and Good, Benjamin M. and Saravia-Butler, Amanda M. and Nelson, Charlotte A. and Balhoff, James P. and Kebede, Yaphet and Whetzel, Patricia L. and Bizon, Christopher and Su, Andrew I. and Baranzini, Sergio E.},
  year          = {2026},
  eprint        = {2605.30283},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2605.30283}
}

Funding

• National Science Foundation Award #2333819: "Proto-OKN Theme 1: Connecting Biomedical information on Earth and in Space via the SPOKE knowledge graph"
• National Science Foundation Award #2535091: "Proto-OKN Theme 2: OKN-Fabric"

License

BSD 3-Clause. See LICENSE.

Copyright (c) 2026, Structural Bioinformatics Laboratory