Docs
Restormel Testing is the suite’s open-source layer for goal-based, workflow-aware tests of AI-enabled software—local and in CI/CD—with model execution through Restormel Keys (BYOK). Follow the walkthrough for the fastest path from zero to a passing (or intentionally failing) run you can debug.
What brings you here?
- Starting a new project Install the CLI, add `restormel-testing.yaml`, first local run.
- Adding Testing to an existing stack Playwright, CI, and Keys already in place—wire the runner without a rewrite.
- CI / GitHub Actions PR checks, verdicts, and reproducing a failed job locally.
- BYOK through Restormel Keys Logical model refs, provider resolution, no parallel secret system.
- Escaping brittle scripted E2E Goals and success criteria for AI-heavy journeys instead of selector soup.
Product and components
- CLI —
init,validate(with optional--json),run(repeatable--suiteor--suites, optional--goal,--json),report,doctor. - Runner — orchestrates goals, retries, verdicts (
passed/failed/indeterminate), and artefacts. - Browser adaptor — Playwright-backed execution for MVP (substrate, not the product headline).
- Keys adaptor — resolves logical model references; aligns test execution with your Keys project. Optional hosted provider keys in Keys reduce CI env archaeology for
judge_rubric. - Reporting —
report.json, Markdown summaries, GitHub step-summary Markdown, JUnit, plus areproductionhint for failed runs. - GitHub Action — thin wrapper around the same run contract as local.
Runtime and CI
One runner contract; different install paths. See Compatibility for the matrix and version targets as they stabilise.
| Target | Status | Notes |
|---|---|---|
| Node.js | 20.x (project engine) | Align with Restormel module default stack. |
| Local CLI | MVP path | Same config as CI; artefacts on disk. |
| GitHub Actions | MVP path | PR annotations and fail-the-job semantics. |
| Other CI | Bring your runner | Invoke the CLI; no hosted platform required. |
Quick links
- Walkthrough — Phase 0 → verification strategy
- How it fits together — Runner, Keys, browser, CI
- Configuration — `restormel-testing.yaml`
- Test definition — Suites, goals, success criteria
- Keys integration — BYOK seam
- Keys + Testing onboarding — Connections, Testing hub, CLI env
- Keys + Testing (from Testing docs) — Links into Keys dashboard
- CI guide — GitHub Actions
- Existing stack (monorepo recipe) — pnpm, Playwright, storage state
- Keys in CI — When RESTORMEL_KEYS_* is required
- Fork PRs & triggers — pull_request vs pull_request_target
- HTTP runs vs Action — No /v1/runs in MVP
- Performance goals — Vitals + Lighthouse paths
- Architecture — Packages and boundaries
- Plot dogfooding — First real suite
- Examples — Reference layouts
- Keys docs (canonical) — Env vocabulary, resolve API
- Restormel Keys — Suite product home
- Repo Testing docs — Markdown specs (canonical drafts)