Restormel Testing — open source

Stop shipping AI flows you can’t regression-test in CI

Restormel Testing is open-source infrastructure for goal-based, workflow-aware tests of AI-enabled software—in local development and CI/CD—with model execution through Restormel Keys (BYOK).

Define suites in repo config. Get verdicts and artefacts you can act on. Not a hosted control plane requirement for the core loop.

  • Goals, not just clicks

    Express critical journeys as outcomes and success criteria—so a run can fail when the user outcome fails, not only when a selector breaks.

  • Keys-backed execution

    Resolve providers and models through the same Restormel layer your app uses. Keep cost and provider choice under your keys.

  • Built for CI

    The same run contract locally and in GitHub Actions—structured summaries, retries, and evidence for triage.

Composes with Playwright for browser automation. Docs and examples track the MVP CLI and restormel-testing config.

Get started Browse the docs →

What brings you here?

What Testing is

A testing layer for teams shipping prompt-driven features, browser flows that depend on AI, tool-using agents, and retrieval-backed behaviour. It is designed for non-determinism: explicit verdicts, retries, and artefacts instead of pretending every run is a simple boolean.

It extends the Restormel suite the same way Keys extends your runtime—open source, repo-native, and meant to run where your pipeline already runs.

Out of scope for this product story:

  • Not a generic Playwright wrapper—browser automation is a substrate, not the headline.
  • Not a generic eval dashboard or prompt playground.
  • Not an observability platform—use your existing production tooling for that job.
  • Not a replacement for unit tests, types, lint, or security scanning.

Built where it ships first

The MVP is intentionally small and dogfooded on real product surface area before the scope balloons.

Plot — web-critical journeys

First suites target Plot’s web-critical goals: a handful of business-meaningful browser journeys with explicit success criteria, runnable locally and in GitHub Actions, with Keys-backed model execution and triage-friendly failure output.

Read product and MVP docs in the repo →

Where Testing fits in your stack

Testing runs in your pipeline and talks to your app like other developer tools. It is not a proxy in front of user traffic and not a replacement for your AI gateway.

Your product Features, UI, APIs, agents—your domain code.
Restormel Testing Suites, goals, runner, verdicts, CI adapters, artefacts.
Automation & execution Browser automation (e.g. Playwright), job runners, local and CI environments.
Restormel Keys Resolve, policy, and BYOK execution alignment for model calls used in tests.
Providers Direct APIs or gateways you already use.
Diagram placeholder — stack illustration can ship here without layout changes.

What it tests

  • AI features and prompts that change behaviour without a traditional “code diff”.
  • Browser and UI flows where AI is on the critical path to the user outcome.
  • Agent-style workflows with tools or multi-step reasoning.
  • Retrieval-aware behaviour where quality degrades before hard errors appear.
  • Regression when models, providers, or prompts change—via Keys-aware execution.

How it works

  1. Config in the repo — suites, goals, environments, and opaque Keys references (never raw secrets in YAML).
  2. CLI or CI starts a run with the same contract everywhere.
  3. Runner executes browser-backed goals with retries, verdicts, and artefact writes.
  4. Keys adapter resolves logical models to provider execution.
  5. Reporter writes summaries and artefacts for local triage and downstream CI consumers.

Built on Restormel Keys

Keys is the suite product for routing, policy, and BYOK. Testing uses that seam so test runs don’t bypass your control plane and don’t invent a parallel credential system. Configure projects and keys where you already do for production workloads.

Restormel Keys · Keys documentation

What’s in the box

Suites, runs, and CI-friendly outputs—composed from mature pieces, with Restormel-owned contracts in the middle.

  • Repo-native suites

    YAML suites and goals, environments, timeouts, and retries—checked into git.

  • Goal-based runner

    Browser-backed runs with structured verdicts: passed, failed, or indeterminate.

  • Keys integration

    Logical model references resolved through Restormel Keys; no second key vault.

  • CLI & CI

    Local commands plus a composite GitHub Action; Markdown summaries, JUnit, and JSON under each run directory.

  • Artefacts

    Summaries, timelines, and evidence you can triage from a failed job—not a hosted observability product.

Add testing to your AI pipeline

Install the CLI, define a small suite, wire Keys, then run the same job locally and in GitHub Actions. Follow the walkthrough from inventory to CI.

Get started Open GitHub →
Media placeholder — screenshot or demo can replace this region.