04 /Exhibit № 04 · AI Evaluation
An AI Evaluation Lab — reproducible, not a chatbot demo.
Provider-neutral model aliases scored against versioned prompts and synthetic datasets across three neutral scenarios. It runs offline in fixture mode with no API keys, and it produces the same numbers every time.
Fixture mode — synthetic data, no live models. These pages render committed, reproducible results from the runnable harness in
benchmarks/ai-eval. Models are provider-neutral aliases; datasets are synthetic and openly licensed. Latency and cost are representative fixtures, never live measurements. No employer or customer data, prompts, or IP appear anywhere.01 /Headline
What the reference run shows
Top overall
100%
Frontier A
Models
3
provider-neutral aliases
Scenarios
3
neutral tasks
Dimensions
6
scored per case
Prompt
v2
version under test
API keys
0
fixture mode, offline
02 /Tasks
Three neutral scenarios
01 · Structured extraction
Pull a fixed set of fields from short public-domain text into a strict JSON object — including a case where source PII must not be extracted.
02 · Retrieval-grounded QA
Answer using only the provided passages and cite them — including a passage that carries an injected instruction a safe model must ignore.
03 · Tool / function-call selection
Choose one tool and emit arguments that validate against its strict schema — routing a destructive action through a confirmation step.
03 /Scoring
Six dimensions
| Dimension | Question | How it's measured |
|---|---|---|
| Correctness | Is the answer right? | Field accuracy for extraction, normalized answer match for QA, and correct tool + arguments for tool selection. |
| Groundedness | Is the answer supported only by the sources? | Citation F1 against the expected passages for QA — an answer that cites the wrong passage or none scores low. |
| Format compliance | Does the output match the required schema? | The output is validated against the scenario's JSON schema (object shape, types, enums, no extra keys). |
| Latency | How fast is the response? | Mean response time recorded in the fixtures. Representative, not a live measurement. |
| Estimated cost | What would a run cost? | Token counts × the alias's illustrative per-token pricing, reported per 1,000 cases. Never billed, never claimed live. |
| Safety | Does it refuse unsafe actions? | Seeded probes: don't extract PII, don't follow injected instructions, don't issue a destructive call without confirmation. |
04 /Method
Methodology & limitations
Every model response is a committed fixture and every scorer is a pure function, so re-running the suite — and regenerating these fixtures — is byte-for-byte stable.
Latency and estimated cost are representative values from the fixtures, clearly labelled and never presented as live numbers. Real provider adapters would plug in behind the neutral aliases.
- Model aliases are capability-tier stand-ins, not specific vendors — the numbers illustrate an evaluation method, not a vendor ranking.
- Datasets are tiny and synthetic; scores show the shape of an evaluation, not a general-capability claim.
- Latency and cost are recorded fixtures, so they reflect a reference run, not your environment or current pricing.
- No live inference runs in the public demo; the runnable harness in benchmarks/ai-eval is where real adapters would attach.