The twelve questions, across three models

SUMMARY.md reads the study by arm. This reads it by question — which questions discriminate between models, which defeat all of them, and which are carrying the aggregate differences.

Every figure here is derived from runs-*-cross/conditions/*.json and can be recomputed from summary.csv in the same directories.

The matrix

Attempts answered exactly, out of 3, as KG named / KG not named / −6 tools:

Row totals are out of 27 — three arms × three attempts × three models.

What the rows say

Opus answers nine of the twelve perfectly, in every arm. Its entire deficit sits in three questions: ncipidkg, prokn and biomarkerkg. The aggregate figure of 99/108 is not a model that is slightly unreliable everywhere; it is a model that is completely reliable on 75% of the corpus and inconsistent on three questions.

Haiku scores zero on three questions across all nine of their attempts — spoke-okn, gene-expression-atlas-okn and biomarkerkg. It is not that these are marginal for Haiku; they are simply out of reach in every condition, which is why the tooling axis has so little to say about it.

biomarkerkg is the corpus's hardest question by a wide margin — 9/27 against the next worst at 13/27, and the only question Opus fails in its best-equipped arm.

What is wrong with biomarkerkg

Eight of the nine model-arm cells record a diagnosis, and all eight are the same one:

0/3 columns align; no counterpart for geneSymbol, accession, pathwayCount

That line is a red herring. Scoring is denotational: column names are ignored, and the arm that scored 3/3 — Opus with the graph unnamed — submitted the same gene_symbol, protein_accession, pathway_count header as every failing attempt. The message appears whenever an answer names its columns differently, correct or not. The failures are in the values.

A polymorphic predicate, and a missing type restriction

The question asks for proteins that are "the target of a key event in an adverse outcome pathway". The property linking an event to its biological target, edam:data_1025, is asserted on two types — and only one of them is a key event:

A key event relationship is an edge between two key events, not a key event. An agent that anchors on data_1025 without restricting the type picks up both, and then reaches AOPs the protein's key events do not belong to.

That is the entire divergence. Counting distinct AOPs for IL6:

The four extra AOPs are reached entirely through key event relationships. has_key_event is self-protecting — all 1,469 of its subjects are of type KeyEvent — whereas dcterms:isPartOf is generic containment and holds for key events, key event relationships and stressors alike.

The two Opus attempts differ by exactly one line: the passing query has ?ke a aop:KeyEvent, the failing one does not.

The answer key is correct and the prose is already specific. This is not an ambiguous question; it is a trap in the data modeling, and the benchmark caught three models falling into it. Refining the prose to warn about the predicate would mean describing the schema in the prompt, which is the exploration the benchmark exists to measure.

The agent's error compounded it

Opus's failing attempt ranked candidates with the untyped isPartOf path (IL6 = 56), pruned to the top 13 on that ranking, then re-ran with has_* and got IL6 = 52 — the key's number. But CTSB and SREBF1, which place 9th and 10th under the correct relation, had already been eliminated by the wrong-metric prune. It then submitted the earlier table anyway.

So the recorded failure is a type-restriction error, followed by filter-then-verify on the wrong metric, followed by submitting a superseded result. All three are agent failures, and all three are the kind this benchmark is meant to surface.

What the columns do not say

Per-question cells are three attempts. A single cell moving from 3/3 to 1/3 is one or two episodes. Individual flips should not be read as effects; only the row and column patterns are worth much.

F1 below is partial credit over an answer's rows as a multiset — precision = matching rows ÷ rows returned, recall = matching rows ÷ rows in the key, F1 = 2PR/(P+R) — with IRIs compared by local name and decimals at 6 significant digits. Full definition in SUMMARY.md.

The clearest example is Opus on biomarkerkg: 1/3 with the graph named against 3/3 without it, which looks like an inversion — naming the graph making things worse. The per-attempt F1 values are all-or-nothing (1.0, 0.0, 0.0 against 1.0, 1.0, 1.0), so this is a binary flip on a question that is unstable for everyone, not evidence that the hint hurts.

Difficulty is only loosely shared across models. Ranking the twelve questions by total score within each model and correlating the rankings gives Spearman ρ of 0.43 (Opus~Sonnet), 0.31 (Opus~Haiku) and 0.46 (Sonnet~Haiku). Those are positive but weak, and at n = 12 the confidence interval is wide enough that they should be read as "the models do not obviously agree on what is hard" rather than as a measured degree of agreement.

The one thing all three agree on is the bottom: biomarkerkg is each model's worst question. The top is less clean — biobricks-aopwiki is in every model's best three, but Opus ties nine questions at a perfect 9/9, so it has no meaningful ranking at the top to compare.

Reproducing this table

from pathlib import Path
from okn_benchmark.report import load_conditions

for label, root in (
    ("Opus", "runs-opus-cross"),
    ("Sonnet", "runs-sonnet-cross"),
    ("Haiku", "runs-haiku-cross"),
):
    for c in load_conditions(Path(root)):
        for q in c["questions"]:
            n = sum(1 for a in q["attempts"] if a.get("ordered_exact"))
            print(label, c["_dir"], q["id"], f"{n}/3")

ordered_exact is the stricter of the two match criteria — right rows and right order. Across this study it differs from exact on 3 of 324 attempts — one Sonnet, two Haiku — each a case where the right rows came back in the wrong order.