What is an MCP server's tooling worth, and to which model?

The benchmark measures how well an agent turns a prose question into a correct SPARQL answer over the Proto-OKN federation, and how much of that success comes from the server's tooling rather than from the model. This study asks the follow-up: the tooling is worth something — but worth it to whom?

Three models — Opus 5, Sonnet 5, Haiku 4.5 — answer the same twelve cross-graph questions under the same three conditions, three attempts each: 324 episodes, none blocked.

The six withheld from the ablated arm are get_description, get_query_template, get_schema, list_graphs, route_query and visualize_schema — the tools an agent would use to find out what a graph contains before querying it. Withholding is enforced at the proxy, so a remembered tool name cannot be called anyway.

Each model ran at its own default reasoning effort — high for Opus, medium for Sonnet, and the CLI default for Haiku, which was run without an --effort flag — so this compares the three as they come configured.

The headline: the three models fail in three different places. Opus is nearly flat across all three conditions; Sonnet collapses when the graph is not named; Haiku never reaches the point where the tooling could help it.

Every figure and table here comes from runs-opus-cross/, runs-sonnet-cross/ or runs-haiku-cross/, and can be traced to summary.csv (one row per attempt) or conditions/*.json (settings, provenance and aggregates) in those directories. The same study read question by question is in QUESTIONS.md; how the figures are built and rebuilt is in README.md.

1. Methods

Corpus. 12 .rq files (benchmarks-cross/ in this repo), all on the cross tier. Each pairs a prose #+ summary: — the prompt given to the agent — with a SPARQL answer key. The summary states every limit and tie-breaker, so the result set is fully determined by the prose. The summary never names the knowledge graph; choosing it is half the task.

Knowledge graphs. Each question joins a different pair, so twelve joins are exercised:

The ablated arm withholds six discovery tools from an otherwise-full toolkit: get_description, get_query_template, get_schema, list_graphs, route_query, visualize_schema. Withholding is enforced at the proxy on both halves of the protocol, so a remembered tool name cannot be called anyway, and every refusal is recorded.

Why there is no baseline arm. The obvious control would be a SPARQL-only arm — one neutral query tool, no server, no instructions. It cannot be used here, because it cannot be run with the graph unnamed. An agent has no way to address the federated endpoint without the list of valid graph names, and once it has that list it recognizes the graph a question is about rather than having to work it out. The control would therefore sit permanently on the "KG named" side and could never provide the other half of the comparison.

None of the comparisons need it. Full toolkit against ablated toolkit measures what the six tools add; graph named against graph not named measures what naming it saves. Both are differences between arms that exist.

Agents. Three attempts per question, driven through claude -p 2.1.234, with a 900s per-episode timeout:

Each model ran at its own default reasoning effort — high for Opus, medium for Sonnet, and the CLI default for Haiku, which was run without an --effort flag. The comparison is therefore between the three as they come configured, rather than at a single effort level held across them. The figures label these effort high, effort medium and effort default.

Isolation. A proxy sits between the agent and the server: it filters the tool set, strips or forwards the server's instructions, injects the answer channel and times every call. Each episode runs in an empty scratch directory with no settings sources, no skills and no delegation tools, so nothing but the tools under test can reach the data.

Ground truth. Answer keys are executed against the Proto-OKN federation endpoint; the rows are cached and fingerprinted per question, so editing a key invalidates only that question.

Execution. Strictly serial — timings are meaningless under contention.

2. How answers are scored

Answers are compared by denotation, not by SELECT clause: column names and column order are ignored, so an answer is free to name its variables however it likes. Where it does use the reference's column names, values are matched column by column, which catches a row that swaps two same-typed fields. Two further adjustments make comparison fair across engines — IRI normalization to local names, and a numeric tolerance for computed decimals.

Four numbers appear below and must not be conflated:

• always exact — questions that matched on every attempt. Deliberately harsh: a question passing two attempts of three counts as zero.
• ever exact — questions that matched on at least one attempt. The gap from always is flakiness, which is what repeats exist to expose.
• success rate — for each question, the fraction of attempts that matched, averaged over questions.
• mean F1 — partial credit, so a near-miss scores above zero. Computed per attempt over the answer's rows, treated as a multiset so duplicate rows count:

precision = matching rows ÷ rows the answer returned recall = matching rows ÷ rows the key returned F1 = 2 · precision · recall ÷ (precision + recall)

A row matches only if it agrees cell for cell, after the same two adjustments used for exactness — IRIs compared by local name where that is unambiguous, and decimals compared at 6 significant digits with a 0.0005 relative tolerance. When the answer uses exactly the reference's column names, cells are matched column by column; when it renames anything, the whole row is compared by denotation instead. Both sides empty scores 1.0; no overlap scores 0.0.

The reported figure is a mean of means: F1 is averaged across a question's attempts first, then across questions. So every question weighs the same regardless of how many attempts ran, and a question is not counted twice for being flaky.

Attempts exact, below, is ordered_exact — right rows and right order, the stricter of the two criteria and the one the figures' totals count. The order-insensitive exact is one higher for Sonnet and two higher for Haiku.

3. What each attempt produced

Two of the three conditions are degraded on purpose, so a success rate pooled over all 324 attempts would not describe anything. The equipped arm is the baseline to read: 76 of its 108 attempts were fully correct, from 34/36 for Opus down to 13/36 for Haiku.

What the pool does show is the shape of failure. Of the 122 attempts that were not fully correct, 94 returned wrong rows, 25 were killed at the timeout, and only 3 got the right rows in the wrong order. Ordering is almost never what goes wrong — which is why the strict and order-insensitive scores differ on 3 attempts out of 324.

The bands separate by model far more than by condition. Opus's worst arm (32/36) beats Sonnet's best (29/36), which beats Haiku's best (13/36) by more than a factor of two. Whatever the tooling is worth, it is worth less than the gap between these models.

Read down the always exact column and the three models tell three different stories:

• Opus barely moves. 11 → 10 → 9 across conditions, and ever exact stays at 12/12 in every condition — there is no question it cannot do without the tools, only questions it does less reliably. The tooling buys consistency, not capability.
• Sonnet is the model the KG hint matters to. Not naming the graph costs it 4 questions (7 → 3) and 8 attempts (29 → 21) — the largest single effect anywhere in the study. Withholding the six tools costs it less, one question.
• Haiku never gets far enough for the question to apply. 1/12 in its best condition. Its ever exact of 8/12 shows the questions are not beyond it in principle, but nothing is reliable, and the two deltas (−1 and −1 question) are inside the noise of a 12-question set.

The hollow squares concentrate too: Opus has none at all in 108 episodes, against 13 for Sonnet and 12 for Haiku. Stalling is a model property here, not a condition one.

4. Where the time goes

The third row of each band reddens visibly, the second only slightly. Both ablations raise endpoint load in every model, but by very different amounts — and the median attempt is the honest comparator here, since row totals are distorted by a handful of very long episodes:

Not naming the graph costs about 1.4× in every model; withholding the six tools costs 4.5–9.5×. Whatever the agent does to recover a graph it has not been told about, it does not do it by hammering the endpoint — that comes later, in Figure 4.

Opus's equipped arm is the bluest region of the figure: a median attempt spends 6.3 seconds at the endpoint, and the whole corpus takes 501 seconds. That is roughly an eighth of what the same model needs without the tools, bought for a two-question accuracy drop — the sharpest illustration in the study that the tooling buys efficiency rather than capability — while nothing is rationed. Section 9 rations it, and the distinction does not survive.

The baselines rise as the models weaken: Haiku's equipped median is already 3.9× Opus's. A weaker model spends more at the endpoint before anything is taken away.

5. How much work reached the endpoint

Query counts rise when the tools go — Opus 344 → 533, Haiku 541 → 940 — but nowhere near as fast as server time did. Dividing one figure by the other gives the weight of an average query:

Opus's queries become 4.8× more expensive each. It is not issuing more of the same work: without get_schema and the router it writes broader, heavier queries — rediscovering the shape of the graph by scanning it.

Every arm ran zero failed queries. Whatever else the ablation changes, it does not make these models produce invalid SPARQL.

6. What it cost

Cost does not track server time. For both Opus and Sonnet the costliest arm is KG not named, while the slowest arm at the endpoint is −6 tools:

Haiku is omitted because for it the two coincide, and its three totals are near-identical lower bounds.

Per attempt, Haiku costs $0.25–$0.33 against Opus's $0.74–$1.23. But it answered 28 attempts correctly to Opus's 99, in twice the wall-clock time. Over the whole study Opus cost $105.02 and Sonnet ≥$100.01, for 99 and 75 correct attempts. The cheap model was not a cheap way to run this benchmark.

7. Totals

Per arm, in absolute terms — the same numbers the figures' row totals carry:

≥ marks a row containing an episode killed at the 900s timeout: its server time stopped early and its cost was never reported, so both understate, and they understate most in the arms that struggled most.

Per model, across all three arms:

Opus answered 3.5× as many attempts correctly as Haiku, in half the wall-clock time, for three times the price. Sonnet cost essentially the same as Opus for three-quarters of the accuracy and twice the time. A smaller model was not a cheaper way to run this benchmark — only Haiku was materially cheaper, and it was the one that mostly did not work.

8. Which tools were used

query, get_join_strategy and submit_csv appear in every arm. route_query and list_graphs appear almost exclusively when the KG is not named, which is the KG-hint axis behaving as designed. get_join_strategy is used throughout — this is the cross tier, where a join always has to be worked out.

9. Capability under a resource budget

Sections 4 to 7 report both ablations as costs in time and money, and section 3 reports almost none in accuracy — Opus's ever exact never leaves 12/12 whatever is taken away. Those two readings coexist only because nothing here was rationed: the single ceiling was the 900s agent timeout, which bound on 25 of 324 episodes and on none that answered correctly. Under a budget, efficiency is capability — and the study can be asked that question without running anything, because what each attempt spent is already recorded per episode. Below, an attempt counts only if it was correct and stayed inside the budget.

Nothing in this section is a lower bound. It reads the resource use of correct episodes only, and an episode killed at the timeout produced no answer — a failure at every budget, never in a numerator. The ≥ that qualifies the totals in section 7 has no counterpart here, and build_budget.py refuses to print at all if any correct episode has an incomplete record.

Under a server-time budget

Ration the endpoint and the ablated arm collapses where uncapped it looked nearly free. Opus falls from 32 correct to 5 at a 20-second budget, while its equipped arm only goes 34 → 29: a two-attempt difference becomes a twenty-four-attempt one. The KG-hint arm barely notices (33 → 21), because it was never spending its resource here.

One panel per model, all six sharing one pair of axes so a difference in shape is a difference in the data and not in the scale. Each line steps up once per correct answer, ordered by what that answer cost, and flattens at that arm's uncapped total. So a point reads: at this budget per attempt, this arm would still have got this many right. The curves are exact at every budget, where the tables above sample four — and a table of four columns cannot show where two lines cross.

The ablated arm is shifted right in all three panels, and it is the same shift each time. That is what makes it a property of the toolkit rather than of a model: green tracks blue in shape and lags it in price, on models whose uncapped scores differ by a factor of two and a half. The KG-not-named arm sits between them, nearer to equipped than to ablated — withholding the graph name costs endpoint time, withholding the six tools costs an order of magnitude more of it.

Under a cost budget

Ration tokens instead and the ordering reverses in the two models with room to show it. At $0.50 Opus scores 18 equipped and 8 with the six tools withheld, but only 3 with the graph unnamed; Sonnet is 15 equipped, 13 ablated, 3 unnamed. The arm that survives an endpoint budget is the one a token budget breaks. Haiku does not reverse — its ablated arm is the worse of the two on both axes — but Haiku is the model section 10 reports as saturated, and its uncapped 13 / 8 / 7 is already inside the noise of a twelve-question set.

The two ablations swap places between the two figures. Read the panels side by side: under a server-time budget green (−6 tools) is the laggard and orange (KG not named) is close to the baseline; under a token budget it is the reverse, with green tracking blue closely and orange falling away. Same nine arms, same episodes, opposite conclusions — the only thing that changed is which resource is scarce.

This is the "two currencies" result of section 10 restated as capability rather than as overhead: whether withholding a tool costs you answers depends on which resource you are short of.

Budget needed to reach 50% of the equipped arm's uncapped score

Reading a single fixed budget invites picking a flattering one, and the tables above span an eight-fold range in server time and a six-fold one in cost — enough room for a reader to find almost any headline. Inverting the question resists that: for each model, the smallest budget at which an arm still gets half of what the equipped arm managed with no budget at all.

The server column separates cleanly, and the same way in all three models — withholding the six tools costs 8.9–13.2× the endpoint budget, withholding the graph name 1.5–3.7×. The cost column is both smaller and inconsistent in direction (1.8× / 3.7× / 1.5× against 1.6× / 1.1× / 3.4×), so the token-budget reversal above is a two-model observation and not a law. The endpoint-budget result is the one that carries.

This budget was applied after the fact

Every number above is censoring, not a run under a constraint. Each episode was recorded with no budget in force, and a budget is then drawn across what it happened to spend. That assumes the agent's trajectory would have been identical up to the moment the budget bound — which is exactly what the 900s timeout does, and exactly what a real budget would not do.

The assumption fails in a known direction. An agent told it has twenty seconds of endpoint time would triage: fewer exploratory queries, narrower patterns, an earlier commit to an answer. It should therefore score better than truncating its unconstrained trajectory predicts. Read the capped columns as a floor on the capped arm's score, and the gaps as a ceiling on the effect. What this establishes is that the effect exists and which resource it is denominated in — not its size.

Running it prospectively

The harness is most of the way there. --agent-timeout is already a prospective wall-clock budget, and the proxy already times every call, so a cumulative server-time budget is the same mechanism one level down: refuse once an episode's endpoint total passes the ceiling, and record the refusal the way a withheld tool's is recorded, so a budget-exhausted episode is distinguishable in the log from a wrong answer. A token budget is harder and would not be symmetric — the agent host reports cost only on a clean exit, so it would have to be approximated from token accounting mid-episode rather than enforced exactly.

Two design points would matter more than the mechanism. Whether the agent is told its budget decides what is being measured: an unannounced ceiling reproduces the censoring above under real conditions, while an announced one measures triage, which is the more useful capability and the harder scoring problem. And the budget has to be fixed across arms, not per-arm, or the comparison dissolves. Both make it a fourth axis rather than a fourth condition, and axes are expensive: three budget levels across the three existing arms is nine cells, 324 episodes per model — the whole study again, three times over. A single ceiling added to the three arms is 108 episodes, and answers the narrower question of whether the censoring above survives enforcement.

10. Overall assessment

The two ablations stress different resources

Across all three models the two axes separate cleanly — and they separate the same way in each one, which is what makes the result trustworthy at n = 12.

Values are Opus / Sonnet / Haiku.

Withholding the KG name is a reasoning cost. Token cost rises in all three models (1.28–1.65×) while endpoint work barely moves. The agent has to work out which two graphs the question is about, and that happens in the model rather than at the endpoint.

Section 9 restates this as capability rather than overhead. Give an agent a fixed endpoint budget and the ablated arm starts losing answers it does not lose here; give it a fixed token budget and the unnamed-graph arm does instead. Which tool is worth having depends on which resource is scarce — a question this study can only ask retrospectively, and section 9 says so.

Withholding the six discovery tools is an endpoint cost. The median attempt gets 4.5–9.5× slower at the server in all three models, while token cost stays essentially flat (0.97–1.29×). The agent compensates for the missing schema by interrogating the graph directly.

So an ablation that looks mild on one axis can be severe on the other, and neither time nor money alone describes what a tool is worth. get_schema and the router are cheap in tokens and enormous in endpoint load; naming the graph is the reverse.

What running the ablation on weaker models adds

The intuitive expectation — that a weaker model leans on the tools more, so the ablation shows a larger effect — is only half right, and the half that fails is the half that matters for measurement.

The attempt-level signal does grow as the model weakens. Removing the six tools costs Opus 2 attempts, Sonnet 4 and Haiku 6. On that measure the weaker the model, the more the tools are demonstrably worth.

The question-level signal collapses. On always exact the same ablation costs −2, −1 and −1. Haiku starts at 1/12 and cannot fall below 0/12, so the metric saturates before the effect can be seen; its two ablation deltas are indistinguishable from each other and from noise.

And for Haiku the failure sits upstream of the tools. Its ever exact is 8/12, 5/12 and 7/12 — it does not reliably reach a correct answer in any condition, and three questions are blank across all nine of their attempts. What a tool would have saved a model that is not succeeding either way is not measurable.

Sonnet is the informative middle case: capable enough for the metric to have room to move, weak enough that it does move. The KG-name ablation costs it 8 attempts and 4 questions — the largest single effect in the study, and one Opus is too strong to show.

A tooling ablation therefore needs a model that can already do the task. Too strong, and the accuracy deltas are small while the real result moves into efficiency, as it does for Opus. Too weak, and there is no signal at all. Report attempt-level counts alongside question-level ones, or the weak-model arm will read as saying nothing when it is in fact where the raw counts move most.

11. Limitations

• The budgets in section 9 were applied retrospectively. No episode was run under a constraint; each was recorded unconstrained and then censored at a budget. A budget-aware agent would triage rather than be truncated, so those capped counts understate what a real capped run would score. The section states the direction of the error and what running it prospectively would take.
• Small n. Twelve questions, three attempts. One question flipping is worth 8.3 points — more than several of the differences reported here. The Haiku deltas in particular are inside that noise.
• A single run, not an independent replication. Every cell is three attempts of one pass.
• One question concentrates a single failure mode. biomarkerkg — the corpus's hardest, and the only question Opus fails in its best arm — turns on one type restriction: the property linking an event to its target is asserted on both key events and key event relationships, and an agent that does not restrict the type over-counts. The answer key and the prose are both correct; the models fell into the same trap. Traced in QUESTIONS.md.
• Cost and server time are lower bounds wherever an episode stalled: 25 of 324 episodes were killed at the timeout, and agent.json is only written on a clean exit.
• One server, one corpus, one tier, from a local checkout rather than the deployment.

12. Reproducing the numbers

export CORPUS=benchmarks-cross
uv run okn-benchmark reference --corpus "$CORPUS"          # ground truth, no model

for model in claude-opus-5 claude-sonnet-5 claude-haiku-4-5-20251001; do
  out=runs-${model}-cross
  # full toolkit, both KG-hint modes
  for hint in tags none; do
    uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools all-tools \
        --kg-hint "$hint" --out "$out" --profile profiles/mcp-proto-okn.local.toml \
        --model "$model" --repeat 3 --agent-timeout 900
  done
  # the same toolkit minus the six discovery tools
  uv run okn-benchmark run --corpus "$CORPUS" --tier cross --tools all-tools \
      --kg-hint tags --out "$out" --profile profiles/mcp-proto-okn.local.toml \
      --model "$model" --repeat 3 --agent-timeout 900 \
      --without-tool get_description --without-tool get_query_template \
      --without-tool get_schema --without-tool list_graphs \
      --without-tool route_query --without-tool visualize_schema
  uv run okn-benchmark report "$out" --markdown "$out/README.md"
done

uv run python results-cross/build_comparison.py                  # the six cross-model figures
uv run python results-cross/build_budget.py                      # the section 9 budget tables

Add --effort high for Opus and --effort medium for Sonnet to match the recorded runs; Haiku takes no --effort flag, so it runs at the CLI default. Each is that model's default.

Where the figures live:

Each conditions/*.json records the corpus and its ref, the profile, model, effort, agent version, reference endpoint and engine, and the scoring parameters — so every number is attributable to a specific build rather than to "the server" in the abstract.