Cross-model MCP eval scorecard

How well do different client models actually drive Honua's MCP surface? This page is the answer, published rather than asserted. Every observed result below is rendered from a committed run artifact under mcp/evals/runs/ — the same JSON the eval harness wrote, admin release-readiness is read from mcp/evals/admin-family.v1.json and stays blocked until a live candidate receipt exists. Run artifacts carry the surface they ran against, how they authenticated, and (where the artifact is new enough to record it) the negotiated MCP protocol version and the git SHA of the suite that produced it. Nothing here is hand-typed, and the generator recomputes every rate from the per-scenario rows before publishing it, so a summary that disagreed with its own graded evidence would fail the build instead of reaching this page.

Observation window: 2026-07-05 → 2026-08-24 (5 distinct observation dates, 3 cross-model eval artifacts, 7 certification artifacts).

This is a small, honest corpus, not a benchmark leaderboard. Read What this does and does not measure before citing a number from it.

Cross-model leaderboard

One row per model per distinct observed result. The deterministic control is listed first: it makes no model calls, runs the identical corpus through the identical catalog, and is the CI gate — it is the ceiling the models are measured against, not a competitor.

Column definitions, because these words are used loosely elsewhere:

• Runtime — how the model was invoked. bedrock means the run went through AWS Bedrock; deterministic is the scripted zero-LLM control.
• Passed — the graded workflow met every criterion: required tools called, expected tool order respected, forbidden (mutating) tools avoided, and any asserted answer content present.
• Clarified — the driver ended the run by asking a clarifying question instead of completing the graded workflow; the grader records the violation driver requested clarification. Tracked separately from a failure, and not counted as a pass. The corpus deliberately contains a scenario that requires a clarification round-trip and passes when the model completes it, so asking is a non-pass only where the workflow was meant to continue.
• Errored — the run did not finish; for example, a driver that exceeded its tool-use iteration budget. Also not a pass.
• Tool-error scenarios — scenarios in which at least one tools/call returned an error, whether or not the workflow ultimately passed. In the raw artifacts this ratio is stored under the historical field name editRate; it counts erroring tool calls, not edits.
• Runs — how many committed artifacts recorded this identical result. A model whose result changed between runs appears as more than one row.
• Suite SHA — the commit of the eval suite that produced the artifact, so a row can be re-run. not recorded means the artifact predates the provenance block (see Provenance).

Per-scenario matrix

Legend: ✅ pass · ❌ fail · ❓ clarified · ⚠️ error · · not run

operator corpus

Surface: https://demo.honua.io/mcp · observed 2026-07-05, 2026-07-07

Every non-passing run

A wins-only scoreboard is marketing. Every non-passing graded run in the committed corpus is listed here, with the grader's own violation text.

Read these as capability signals, not bugs in the surface: the eval grades whether a client composed the right workflow, so a smaller model that asked for clarification it did not need, or looped past its iteration budget, is exactly what the corpus is designed to expose.

Admin operation family

The generated Admin REST client covers 396 operations: 385 are published as Admin MCP tools and 11 one-time-secret/session operations are explicitly excluded. The default server roster is 432 tools = 47 static + 385 Admin MCP. The row stays blocked until that exact paginated roster is certified against the same release candidate through both HTTP and the proxy.

Evidence definition: mcp/test/certification/admin-parity.test.ts. REST source: f897700159e2791c9468c6ca85bb4e2a3a8d8433 / edbbef2c19d2730f2c87c0641e189ae9fa83c49f38e29eb40057789ade11555a. MCP coverage: config/admin-mcp-coverage.v1.json / 0b24f61feefe18177e0abc76c491b2c86827b30a5dd45092a595cd595376f088; exclusion roster d93bdf6c31e6c532d5483b08315fed5decdd8f5cc56900e59e45be2eddb2fb6f. Reviewed server head: c810ef3df29269527d4eceb26151921c8c5d5eab. Final server contract head: review-head-validated-awaiting-merged-trunk-pin. This is a readiness row, not a fabricated live pass receipt.

Protocol certification (zero-LLM control)

Separate from the model eval, a deterministic certifier checks the same live surface against the vendor-neutral geospatial-MCP standard: tool schemas, structured output, error shape, pagination, and the standard tool families. It makes no model calls, so it is free to run on a schedule and it fails loudly.

Certification failures and skips — 2026-08-24

From mcp/evals/runs/2026-08-24/cert-standalone.json. Failures are real conformance defects in the certified surface, published unedited; skips name the reason they could not be checked.

The same run recorded 39 known standard gaps — tool families in the geospatial-MCP standard the certified surface does not yet advertise (Analysis and geoprocessing (reference shape), Analysis verbs, App composition, Composition review (reference shape), Control-plane proposal (reference shape), Discovery and grounding (Honua extension), Discovery and query (reference shape), Execution, Execution (reference shape), Feature editing, Intent and planning, Map composition, Publishing, unclassified). They are enumerated in mcp/evals/runs/2026-08-24/cert-standalone.json under knownGaps.

What this does and does not measure

It measures whether a client model, given a plain-language GIS goal and nothing but the MCP catalog the surface advertises, composes the right workflow: discovers layers before querying them, grounds an ambiguous goal before planning it, validates a plan before a dry run, and — critically — stays read-only when the task is read-only. Mutating lifecycle tools are forbidden by the grader on the read-only scenarios, so "did the agent stay inside its authority?" is a scored criterion rather than a hope.

It does not measure general model quality, latency, cost, or output prose. It is not a head-to-head model benchmark: the rows above were observed on different dates against a live surface that changed underneath them (the advertised tool count across these artifacts ranges from 15 to 20). Treat a one-scenario difference as noise; the deterministic control is the only row whose meaning is stable across dates.

Other honest limits:

• Corpus size. The published corpus is 8 operator scenarios. That is enough to surface gross workflow failures and nowhere near enough for a confidence interval. No statistical claim is made or implied.
• Single run per (model, date). Rates are not averaged over repeated sampling, so a rate describes the recorded runs and is not an expectation.
• Surfaces covered. Every published row targets https://demo.honua.io/mcp. The platform-free standalone corpus (a plain public FeatureServer, semantically graded) has no committed cross-model artifact yet, so it is absent from this page rather than summarised from memory.
• Paid lane. Live cross-model runs bill real model usage, so they are dispatched manually rather than on every commit. That is why observation dates are sparse.

Provenance and reproducibility

Every artifact behind this page, with the surface it targeted and the suite that produced it.

3 of the artifacts above predate the self-proving provenance block and therefore carry no suite SHA or negotiated protocol version. They are published as-is, labelled not recorded, rather than back-filled — a provenance field that was never observed is not a field this repo will invent.

An unresolved required tool would mean a scenario failed because the surface never advertised the tool, not because the model chose wrongly. The eval records that separately, per artifact, as catalog.unresolvedRequiredTools.

Across every published artifact that list is empty, so every non-pass above is a genuine workflow-composition result rather than a missing tool.

How to reproduce, and how this page stays honest

Run the deterministic control locally — free, offline, no credentials, no model calls:

npm ci && npm run build            # build the SDK the MCP package consumes
npm ci --prefix mcp
npm run --prefix mcp eval:offline  # deterministic control over the fixture surface

Be precise about what that reproduces: it grades the control against the offline fixture surface, not the live operator surface the rows above targeted. The published control rows ran against the live /mcp — still with zero model calls — so reproducing one needs the live lane below with --driver left at the control.

Re-run a live cross-model row (billable; needs credentials and a reachable /mcp):

HONUA_MCP_REMOTE_URL="https://demo.honua.io/mcp" \
HONUA_API_KEY="$HONUA_DEMO_API_KEY" \
HONUA_EVAL_REQUIRE_AUTH=1 \
HONUA_EVAL_BEDROCK=1 AWS_REGION=us-west-2 \
HONUA_EVAL_BEDROCK_MODEL="us.anthropic.claude-sonnet-4-5-20250929-v1:0" \
  npm run --prefix mcp eval:live -- --driver bedrock

Then commit the artifact under mcp/evals/runs/<YYYY-MM-DD>/ and regenerate:

npm run docs:mcp-scorecard         # rewrite this page from the committed artifacts
npm run docs:mcp-scorecard:check   # the CI gate: fails if the page drifts from them

The check mode runs in the docs-site pipeline next to npm run verify:llms and npm run docs:comparison:check, and mirrors their derived-artifact policy: strict on trunk, relaxed on pull requests (where it still proves the page is producible from the committed evidence). The workflows that commit a new run artifact — the scheduled certification and the manual live cross-model lane — regenerate this page in the same commit, so trunk stays fresh without wedging open PRs. Because the renderer reads no clock and no network, the strict check has exactly one failure mode worth having: the committed artifacts and the published page disagree. Adding a run without republishing, or editing a figure on this page by hand, both fail it.

The relaxed mode is not a hole. It still loads and validates every artifact and re-renders the page, so an artifact whose model summary contradicts its own graded rows, whose schema version is unsupported, or that grades a scenario absent from its declared corpus, fails CI everywhere, on every branch. Only the committed-freshness comparison is deferred.

Related evidence

• mcp/evals/README.md — the evidence corpus, grading taxonomy, and how runs land in the repo.
• mcp/README.md — the MCP server itself, the eval CLI, and the live-lane environment contract.
• mcp/src/eval/operator-corpus.ts — every scenario prompt and grading criterion in the operator corpus.
• Coding-agent evaluation scorecard — the sibling measurement: can a coding agent write correct SDK code on the first try?
• How Honua compares — the same generated-evidence discipline applied to bundle size, protocol coverage, and time-to-first-map.