From a live test console to a deterministic benchmark
Three iterations of the same idea — the last one is the one you can trust in public.
benchmarks/testora.Three iterations
| Stage | Stack | Core idea | Where it hit a wall |
|---|---|---|---|
| v1 — Test console | TestCafe · .NET API · SignalR · Vite/React | One-click E2E runs against real apps with live results streamed over SignalR. | Required a live backend and a running target; results were tied to a specific environment and hard to reproduce or publish safely. |
| v2 — Orchestrator rewrite | Next.js · Drizzle · custom test-engine | Model requirements → suites → fixtures → cases in a database; generate and run scenarios; render HTML/JSON reports. | Strong data model and reporting, but still execution-centric and stateful — not something you can expose publicly without a runner. |
| v3 — Public benchmark (this) | Playwright · offline sample app · committed fixtures · read-only demo | Invert the problem: fix the system under test, seed known defects, and measure detection deterministically. Ship the evidence, not a runner. | Deliberately cannot execute user code. Live/authenticated runs are out of scope for the public surface. |
How the benchmark is built
A pure static app (benchmarks/testora/sample-app) with no network, storage, or timers. Every screen renders from the URL query alone, so a run is a pure function of its inputs.
A catalog (fixtures/scenarios.mjs) declares each scenario's true kind — pass, seeded fail, or controlled flake — and the diagnosis a good suite should produce. It drives both the specs and the scoring.
Playwright runs single-worker with one retry. Seeded regressions fail on every attempt; the flake passes testInfo.retry as its attempt, so it fails first and passes on retry.
A generator validates that live outcomes still match ground truth, then writes byte-stable fixture JSON. The public demo renders that snapshot; CI uploads the real traces as the citable source.
What we gave up, and why
The public surface cannot execute arbitrary code. That closes the door on "run it against your app" demos, but it's the only honest way to publish results without a sandboxed backend.
Committed fixtures are a reproducible distillation, not a dump of every millisecond. Raw millisecond jitter is intentionally excluded so regeneration is byte-stable — the trade is less "liveness" for trustworthy determinism.
Lessons from the legacy system
The SignalR console felt impressive but its results couldn't be re-derived. Fixing the SUT and seeding defects made the benchmark something you can actually reason about.
Counting green tests flatters a suite. Measuring how many known defects it catches, and whether it can tell a flake from a regression, is what maps to engineering value.
Any headline about reduced manual-testing effort is only stated where the supporting source can be cited — the runnable harness and CI in benchmarks/testora — never as an unbacked number. evidence-first