Verification strategy
Canonical verification approach for Restormel Keys integrations — during rollout and ongoing.
This document defines three layers of verification: dashboard checks (visual, no code), CLI checks (terminal, scriptable), and smoke tests (HTTP, automatable). Use all three during cutover (Phase 6) and the first two on an ongoing basis.
1. Verification layers
| Layer | When to use | Who runs it | Automatable? |
|---|---|---|---|
| Dashboard | After any config change (route, policy, credential); during cutover monitoring | Human (or browser automation) | Partially |
| CLI | After install, before deploy, in CI, during cutover | Human or CI | Yes |
| Smoke tests | After deploy, during cutover, as a scheduled health check | CI or cron | Yes |
2. Dashboard checks
Open the Dashboard and verify the following. These checks require no code — just visual confirmation. You can also try the embeddable KeyManager and ModelSelector and run doctor/validate-style checks in the Sandbox (after signing in).
2.1 Project health
| What to check | Where | Expected |
|---|---|---|
| Project exists | Projects list | Your project is listed |
| Environment exists | Project → Environments | production (and staging if used) |
| Gateway Key exists | Project → API Keys | At least one key with rk_… prefix |
| Provider credentials valid | Project → Provider Credentials | Green "valid" status for each configured provider |
2.2 Route health
| What to check | Where | Expected |
|---|---|---|
| Routes exist | Project → Routes | At least one route (e.g. ingestion, interactive) |
| Steps in correct order | Route detail | Steps listed in your intended fallback order |
| Route mode correct | Route detail | fallback_chain (or your intended mode) |
2.3 Policy health
| What to check | Where | Expected |
|---|---|---|
| Policies exist | Project → Policies | At least one policy (e.g. model_allowlist) |
| Policy scope correct | Policy detail | Scoped to the right project/environment |
| No conflicting policies | Policies list | No two policies that contradict each other |
2.4 Usage and logs
| What to check | Where | Expected |
|---|---|---|
| Request count non-zero | Project → Usage | After cutover, requests appear |
| No error spikes | Project → Usage / Logs | Error rate comparable to or lower than pre-cutover |
| Correct route distribution | Project → Usage | Traffic hitting the expected routes |
3. CLI checks
These checks run in a terminal and can be scripted into CI.
3.1 Restormel Doctor (npx @restormel/doctor)
npx @restormel/doctorChecks: Framework detection, packages installed, config file validity, and whether local provider keys are present (if you use them).
Expected: Exit code 0, all checks green.
When to run: After install (Phase 1), before every deploy, in CI on every PR.
3.2 Restormel Validate (npx @restormel/validate)
npx @restormel/validateChecks: Provider access verification for the mode you’ve chosen: gateway-backed (canary request through gateway / auth + reachability) or builder-managed direct (presence/shape checks for required env vars and optional canary calls if you explicitly enable them).
Expected: Exit code 0 if checks pass. Exit code 1 if verification fails.
When to run: Before deploy, in CI on a schedule (e.g. daily), after rotating any gateway/provider credentials in your own infrastructure.
# .github/workflows/deploy.yml
- name: Validate Restormel keys
run: npx @restormel/validate3.3 keys estimate (optional)
npx @restormel/keys-cli estimate gpt-4o --input 1000 --output 500Checks: Returns the estimated cost for a given model and token count.
When to run: Before enabling a new model in a route, to understand cost implications.
4. Smoke tests
HTTP-based tests that verify the resolve endpoint and policy evaluation. Automatable; run after every deploy and on a schedule.
4.1 Resolve smoke test
POST to the resolve endpoint with your project ID, environment, and route ID. Expect HTTP 200 and a non-null providerType (and modelId) in the response. Use RESTORMEL_GATEWAY_KEY for auth.
Expected: HTTP 200, non-null provider in response.
4.2 Fallback smoke test
Create a dedicated test route in the dashboard with a deliberately failing first step (e.g. invalid provider), then a second step (e.g. OpenAI). Call resolve for that route and confirm the result is the fallback provider.
Expected: Resolve returns the fallback step's provider.
4.3 Policy smoke test
Call the policy evaluate endpoint for an allowed model (expect allowed: true) and for a blocked model (expect allowed: false). Use RESTORMEL_GATEWAY_KEY from your backend for auth.
4.4 Combined smoke test script
The script from Phase 6 (Step 6.4) combines all checks:
pnpm run smoke:restormel5. Ongoing monitoring recommendations
| What | How | Frequency |
|---|---|---|
| Resolve latency | Log time per restormelResolve() call; alert if p95 > 200ms | Every request |
| Resolve errors | Count errors from resolve client; alert on spike | Every request |
| Fallback rate | Count fallback-to-legacy or next-step events; alert if > 5% | Every request |
| Credential expiry | Restormel Validate in CI; alert on exit code 1 | Daily |
| Policy violations | Dashboard logs; alert if unexpected blocks | Daily |
| Budget utilisation | Dashboard usage; alert at 80% of cap | Daily |
| Config drift | Restormel Doctor in CI; alert on warnings | Every deploy |
Implementors: See “Agent prompts” below for a ready-to-run CI prompt.
6. Verification schedule summary
| Event | Checks to run |
|---|---|
| Phase 1 complete | Restormel Doctor |
| Phase 2 complete | Restormel Doctor, resolve curl test |
| Phase 3 complete | Resolve with route ID, fallback test |
| Phase 4 complete | Policy evaluate (allowed + blocked) |
| Phase 5 complete | Visual: ModelSelector renders, callbacks fire |
| Phase 6 cutover | Full smoke test, dashboard usage, error rate |
| Ongoing | Restormel Doctor + Restormel Validate in CI; smoke test on schedule; dashboard monitoring |
Checkpoint checklist: mark each section complete as you read it.
Checklist
Prompts for this phase
These are optional and collapsed by default. Use them if you're adding verification and CI gates with a coding agent.