Test definition

Goals express user-meaningful outcomes. Success criteria combine deterministic checks with narrow rubric-backed judgements where needed.

MVP success-criteria shapes include:

  • URL match, DOM signals, text presence or absence.
  • Default bundle is AND. url_matches, dom_signals, text checks, and structured_checks in the same success_criteria object must all pass. For OR composition (e.g. hero or launcher or empty state), use any_of / anyOf with nested criteria objects (see repo docs/testing/config-reference-mvp.md).
  • structured_checks — documented css:… paths; in-page Web Vitals via vital:lcp (and fcp / cls, plus web_vitals: aliases); full category audits via lighthouse:… / lh:… (separate Chrome session — no Playwright cookies). Unsupported paths return indeterminate with a reason code.
  • judge_rubric for tightly bounded cases — needs Keys HTTP resolve or the documented OpenAI fallback; use optional context_selector so the judge sees only a slice of the page (avoid full-page dumps and PII). Purely deterministic suites can omit Keys entirely; see Keys in CI (checklist) and Keys + Testing (dashboard).

Every goal should be business meaningful; avoid exploratory “test everything” goals in early suites.

Verdicts: passed, failed, indeterminate—with reason codes and evidence for CI triage.

Configuration · Performance goals · Plot dogfooding