Configuration

Repo-native `restormel-testing.yaml`: suites, goals, environments, timeouts, retries, and opaque Keys references.

Core concepts (MVP schema):

  • Suite — named collection of goals, environment, timeouts, retries, optional tags.
  • Goal — outcome under test; browser (MVP) or performance; success criteria.
  • Environment — base_url, optional per-env Keys slots, and auth_mode: storage_state with auth_ref (path relative to the config file, or env:VAR for a path to Playwright storage state JSON).
  • Navigation — for each browser (or performance) goal the runner navigates to the environment base_url, then optional per-goal start_path / startPath (relative path under that origin; .. is rejected). Separate environments with different base_url values are still useful when origins differ.
  • Suite hooks — non-empty adapter_hooks (shell commands; teardown runs last), per-goal preconditions, and cleanup are executed by the runner. Opt out with RESTORMEL_TESTING_SKIP_SHELL_HOOKS=1; timeout via RESTORMEL_TESTING_SHELL_HOOK_TIMEOUT_MS (see repo docs/testing/config-reference-mvp.md).
  • Keys — logical slots such as ref:restormel-keys:llm/primary (illustrative); never paste raw provider secrets into this file. You may store encrypted provider keys in the Keys dashboard (Connections) and use Keys + Testing (dashboard) for project IDs and env—CI still uses a Gateway key as RESTORMEL_GATEWAY_KEY (alias RESTORMEL_KEYS_API_TOKEN).
restormel-testing validate --config restormel-testing.yaml
restormel-testing run --suite web-critical --config restormel-testing.yaml
restormel-testing run --suites ci-smoke,web-critical --config restormel-testing.yaml

Canonical MVP field behaviour (executed vs rejected): docs/testing/config-reference-mvp.md. Spec: docs/testing/restormel-testing-mvp-spec.md.

Test definition · Performance goals · Keys in CI · Keys integration