03 /Exhibit № 03 · Benchmark
Testora — an observable test-automation benchmark.
A fixed, offline sample application carries intentional, seeded defects. Testora proves a good suite catches every one of them — deterministically, with complete artifacts — and shows the evidence.
Fixture data, not production. These pages render a committed, reproducible snapshot distilled from a real Playwright run of a seeded sample app. Nothing here is executed live, and no event on this page is a real user event. The runnable harness lives in
benchmarks/testora.01 /Headline
How the reference run scored
Detection rate
100%
2/2 seeded regressions
Flaky identified
Yes
fail-then-pass told apart from a regression
Time to diagnosis
2.3 s
mean across failing scenarios
Artifact completeness
100%
trace · screenshot · video
CI reproducibility
100%
byte-stable across runs
Pass rate
50%
3/6 cases green
02 /What it measures
Five benchmark dimensions
| Dimension | Question | How it's measured |
|---|---|---|
| Detection rate | Does the suite catch every known seeded regression? | Two product defects are seeded into the sample app (un-trimmed email, tax dropped from a total). Detection = share of seeded regressions that end the run failed. |
| Flaky-test identification | Can a genuine flake be told apart from a stable regression? | One scenario is engineered to fail-then-pass across a retry. It must be reported as flaky, not as a passing or a hard-failing test. |
| Time to useful diagnosis | How quickly does a failure become an actionable message? | Mean wall-time across failing scenarios from start to a concise, cause-level diagnostic summary attached to the result. |
| Artifact completeness | Is every failure backed by trace, screenshot, and video? | For each non-passing scenario the run must retain a Playwright trace, a screenshot, and a video. Completeness = captured / expected. |
| CI reproducibility | Do the same inputs yield the same outcomes every run? | The sample app is offline and stateless; outcomes are a pure function of the URL. Re-running the suite — and regenerating these fixtures — is byte-for-byte stable. |
03 /Method
Why the numbers are trustworthy
Every screen renders purely from the URL query (?screen=, ?attempt=). Seeded regressions assert behaviour the app deliberately violates, so they fail on every attempt; the flake passes testInfo.retry as its attempt, so it fails first and passes on retry.
These pages render committed fixture JSON distilled from a real Playwright run. The public demo never executes any test code. The runnable harness and CI upload the real traces, screenshots, and videos as the citable source.