Vionto Studio — an AI pipeline you can trust to fail honestly.
A schema-validated brief-to-render pipeline with explicit human approval gates and idempotent retry. Two stages are seeded to fail — and the benchmark proves the pipeline recovers, truthfully, every time.
fixtures/assets.json). The interactive pipeline below runs the real engine in your browser, with zero network calls. Runnable harness: benchmarks/vionto.How the reference run scored
Five benchmark dimensions
Does every stage produce output that satisfies its schema?
Every stage output is validated against a JSON schema before the next stage can run. 100% means no stage advanced with invalid data.
Can a failed or cancelled run be retried without corrupting history?
A failed or cancelled run is retried and a new job is returned; the original job is never mutated and the new job resumes at the correct stage.
How long does a full brief-to-render run take?
Total wall-clock time from brief acceptance to final render report for successful reference runs, using representative fixture timings.
Does the pre-run cost estimate match what actually happened?
Pre-run cost estimate is computed from the same fixture used at generation time, so delta is zero by construction in this offline benchmark.
Does a deliberately broken stage recover cleanly via retry?
A brief with a deliberately invalid script or transient render failure is retried and reaches succeeded with the same fixture data.
Why the pipeline is trustworthy
The pipeline will not proceed past script generation or past asset-plan generation without an explicit approve() call. A human can also reject at either gate, which ends the run at 'cancelled' — a legitimate terminal state, not a failure — without losing anything: retrying a cancelled run regenerates only the rejected stage.
Retry is legal only from 'failed' or 'cancelled' — the engine throws otherwise. Retrying never mutates the original job; it returns a brand-new job with retryCount+1 that resumes at the stage that failed or was rejected, mirroring the legacy render-job retry route's idempotency rule.
Only a FixtureProvider is wired up — every script, storyboard, and asset plan is a deterministic lookup into a brief's committed fixture. A LiveProviderStub documents the interface a real adapter would implement, but always throws, even with explicit confirmation: no live provider integration exists in this repo.
Limitations
- Five hand-authored briefs demonstrate the method — a happy path, a seeded schema failure, a seeded transient render failure, and two human-rejection paths — not statistical significance at scale.
- "Estimated vs. observed" cost is zero-delta by construction in fixture mode: nothing about a brief varies between estimate-time and generation-time without a live provider. Real variance would only appear once a live adapter is connected.
- The fixture renderer produces a structured report and an SVG storyboard strip, not actual video or image encoding — there is no media pipeline behind this benchmark, synthetic or otherwise.
- Completion-time and cost figures are representative reference numbers, not live measurements — the engine itself is a pure in-memory reducer and runs in sub-millisecond time.