Testing walkthrough

From “I have an AI-enabled product with brittle or incomplete AI journey testing” to “I can run one goal-based Restormel Testing suite locally and in CI using BYOK through Restormel Keys.”

This is the primary onboarding path. It takes you through inventory, install, Keys-backed resolution, your first goal, a local suite run, and GitHub Actions—using the same run contract everywhere. Keep Configuration, Test definition, and Keys integration (or Keys + Testing onboarding) open beside you.

Before you begin

  • A repo with at least one user journey where AI behaviour affects the outcome (prompts, agents, retrieval, model choice).
  • Restormel Keys — a project (or plan for one) if you want BYOK-aligned test execution; optional fallbacks are documented in-repo for early bring-up.
  • Your routing inventory — how the app picks models today and where secrets live (so test config uses logical refs, not a parallel key vault).
  • Toolchain — Node 20.x and pnpm aligned with the Restormel module stack; Playwright available where browser goals run.

Phases

  1. 1. Phase 0 — Inventory Audit AI-heavy journeys; decide goals vs unit/E2E; secrets and Keys posture.
  2. 2. Phase 1 — Install and configure CLI, `init`, `validate`, `restormel-testing.yaml` scaffold, `doctor`; repo layout.
  3. 3. Phase 2 — Wire Keys and confirm resolution Logical model references; prove one resolve path for test execution (BYOK-aligned).
  4. 4. Phase 3 — Define your first goal One business-meaningful outcome with explicit success criteria.
  5. 5. Phase 4 — Run your first suite locally Execute the suite; read verdicts, reason codes, and artefacts.
  6. 6. Phase 5 — Add CI (GitHub Actions) Same run contract as local; composite Action, artefacts, reproduction commands.

Beyond the core path

Same pattern as the Keys walkthrough: after the numbered phases, deeper topics for migration, verification, and secrets hygiene.

  1. 7. Migration paths From Playwright-only scripts, ad hoc eval harnesses, or brittle legacy E2E.
  2. 8. Verification strategy When to trust the gate; flakes; local/CI parity; artefact review.
  3. 9. Secrets and CI setup GitHub Actions secrets, rotation, staging vs prod Keys projects, nightly vs PR jobs.

Implementing with a coding agent

Each phase page includes an Agent prompts section, collapsed by default. Open it when you want an agent to apply that phase to your repository in a bounded, reviewable sequence—mirroring the Keys walkthrough UX.

Keys and secrets at a glance

  • Logical refs in Testing config — point at Keys-managed resolution; never commit raw provider keys in restormel-testing.yaml.
  • Canonical env and API names — Keys vocabulary in Keys documentation; Testing CLI judge env (RESTORMEL_KEYS_*, optional OpenAI fallback) is summarised in Keys integration and docs/testing/config-reference-mvp.md.
  • CI secrets — GitHub Actions (or your runner) holds Gateway keys and project IDs; provider keys may live in Keys (Connections, encrypted) per Keys + Testing onboarding.

Next

Compatibility for Node/CI targets · Restormel Keys product home · New project · Existing stack