Vionto Studio — an AI pipeline you can trust to fail honestly.
A schema-validated brief-to-render pipeline with explicit human approval gates and idempotent retry. Two stages are seeded to fail — and the benchmark proves the pipeline recovers, truthfully, every time.
fixtures/assets.json). The interactive pipeline below runs the real engine in your browser, with zero network calls. Runnable harness: benchmarks/vionto.How the reference run scored
Five benchmark dimensions
Does every stage produce output that satisfies its schema?
Share of schema-validated stage-generation attempts (script/storyboard/asset-plan) that passed validation, across every brief and every retry (14/15 attempts).
Can a failed or cancelled run be retried without corrupting history?
Every retry produces a new, uniquely-identified job rather than mutating the failed/cancelled one, and the engine itself refuses retry from any other state.
How long does a full brief-to-render run take?
Mean representative reference latency across successfully completed runs — the engine itself runs in sub-millisecond time; this figure stands in for a live pipeline's wall-clock time.
Does the pre-run cost estimate match what actually happened?
Maximum difference between the pre-run cost estimate and the cost recomputed from final artifacts. Zero in fixture mode by construction — there is no live provider to introduce real variance; connecting one is where deviation would first appear.
Does a deliberately broken stage recover cleanly via retry?
Share of briefs seeded with a stage failure (a schema-invalid asset plan, a transient render error) that reach 'succeeded' via the documented retry path.
Why the pipeline is trustworthy
The pipeline will not proceed past script generation or past asset-plan generation without an explicit approve() call. A human can also reject at either gate, which ends the run at 'cancelled' — a legitimate terminal state, not a failure — without losing anything: retrying a cancelled run regenerates only the rejected stage.
Retry is legal only from 'failed' or 'cancelled' — the engine throws otherwise. Retrying never mutates the original job; it returns a brand-new job with retryCount+1 that resumes at the stage that failed or was rejected, mirroring the legacy render-job retry route's idempotency rule.
Only a FixtureProvider is wired up — every script, storyboard, and asset plan is a deterministic lookup into a brief's committed fixture. A LiveProviderStub documents the interface a real adapter would implement, but always throws, even with explicit confirmation: no live provider integration exists in this repo.
Limitations
- Five hand-authored briefs demonstrate the method — a happy path, a seeded schema failure, a seeded transient render failure, and two human-rejection paths — not statistical significance at scale.
- "Estimated vs. observed" cost is zero-delta by construction in fixture mode: nothing about a brief varies between estimate-time and generation-time without a live provider. Real variance would only appear once a live adapter is connected.
- The fixture renderer produces a structured report and an SVG storyboard strip, not actual video or image encoding — there is no media pipeline behind this benchmark, synthetic or otherwise.
- Completion-time and cost figures are representative reference numbers, not live measurements — the engine itself is a pure in-memory reducer and runs in sub-millisecond time.