Designing an evaluation you can trust

Each of the three scenarios ships version-controlled inputs and expected outputs. Scoring is a set of pure functions, so "good" is defined before any model runs and the result is reproducible.

Models are capability-tier stand-ins (frontier / balanced / compact). Real adapters plug in behind the aliases; the checked-in fixtures let the whole suite run offline with no API keys.

Correctness alone flatters a model. Groundedness, format compliance, latency, cost, and safety expose where a cheaper tier actually breaks down.

Prompts are code. Because each version is checked in, a prompt change is a reviewable diff whose effect on every score is measurable — including when it makes things worse.

The public demo never calls a model. Latency and cost are representative fixtures, clearly labelled — the trade is liveness for reproducibility and zero-key safety.

Tiny CC0 datasets keep the benchmark legible and IP-clean. They demonstrate an evaluation method, not a general-capability claim — and deliberately contain no employer or customer material.