ai-agent-playwright-typescript-template

Imported from twinklejoshi/ai-agent-playwright-typescript-template (AGENTS.md). Install upstream with npx skills add twinklejoshi/ai-agent-playwright-typescript-template. Copyright stays with the author.

Playwright Test Agents — Architecture & Governance

This document defines the contract, responsibilities, and maintenance rules for the four agents in the AI-driven QA pipeline.

Agent Pipeline Overview

Orchestrator (HITL checkpoints between phases)
    ├─ Phase 1 → Planner (explores app, generates test plan)
    ├─ Phase 2 → Generator (turns plan into executable tests)
    ├─ Phase 3 → Executor (runs tests, collects failures)
    └─ Phase 4 → Healer (debugs failures, patches tests)

Each agent is defined in .github/agents/<name>.agent.md with:

• YAML frontmatter: tools, MCP servers, model
• System prompt: role, workflow, rules

Agent Roles & Responsibilities

1. Planner — playwright-test-planner.agent.md

Role: Explore the live app UI and generate a requirements-driven test plan.

Enforced Conventions:

• Every scenario must map to a REQ-NNN if a requirement file exists
• Every scenario carries a tag recommendation: @smoke (high priority) or @regression (medium/low)
• Scenarios reference locator strategy: prefer data-test IDs, then ARIA roles
• Plans are saved to specs/<feature-name>-plan.md

Startup Sequence (non-negotiable):

1. Call get_framework_conventions → load project POM rules
2. Resolve requirements source in priority order: Jira issue key → requirements file → pasted requirements
3. Convert the selected source payload to canonical structure via normalize_requirements
4. Call planner_setup_page → initialize browser

Key Rules:

• Steps must be specific enough for any tester to follow without ambiguity
• Include negative testing: invalid input, empty states, error handling
• Scenarios are independent and runnable in any order
• Flag gaps: requirements with no discoverable UI flow

Output: specs/<feature-name>-plan.md with clear section headings, numbered steps, tag recommendations

2. Generator — playwright-test-generator.agent.md

Role: Convert plan scenarios into executable .spec.ts files following project conventions.

Enforced Conventions:

1. Always use the fixture, never raw @playwright/test:

   import { test } from "../fixtures/<app>-fixture";

2. Always use page objects, never raw page.* in test bodies:

   await todoPage.addTodo("item"); // ✓
   await page.locator(".new-todo").fill("item"); // ✗

3. Wrap steps in test.step(), never inline comments:

   await test.step("Add a todo item", async () => { ... });  // ✓
   // Step 1: Add a todo item  // ✗

4. Apply tags based on planner's recommendation:

   import { TAGS } from "@utils/configuration";
   test("scenario", { tag: [TAGS.SMOKE] }, async ({ todoPage }) => { ... });

5. Use shared mock data, never hardcoded strings:

   import { TODO_ITEMS } from "shared/mock-data";
   await todoPage.addTodo(TODO_ITEMS[0]); // ✓

6. File header with provenance & traceability:

   // @generated by playwright-test-generator
   // spec: specs/todo-test-plan.md
   // scenario: Add Valid Todo
   // req: REQ-001

Startup Sequence (non-negotiable):

1. Read src/tests/ui/fixtures/ → find fixture for the app
2. Read src/pages/ui/ → find or create page objects
3. Read src/shared/mock-data/ → understand available test data
4. Read src/shared/utils/ → know what helpers exist
5. Read an existing hand-written test → confirm import paths

Key Rules:

• One file per scenario, kebab-case name
• One test.describe per plan section
• Never use page.waitForTimeout or waitForNetworkIdle
• Call generator_setup_page before each scenario
• Call generator_write_test with generated code

Output: src/tests/ui/generated/<scenario-name>.spec.ts with all conventions applied

3. Healer — playwright-test-healer.agent.md

Role: Debug failing tests and apply minimal, targeted fixes.

Enforced Conventions:

• Prefer getByRole, getByLabel, getByTestId over CSS selectors
• Prefer regex matchers for dynamic text
• Never use waitForNetworkIdle or waitForTimeout

Startup Sequence (non-negotiable):

1. Call get_framework_conventions with section "locators" → know preferred selectors
2. Call get_test_health or list_test_health → check heal history

Critical Guard: Healing Limit

• If healCount >= 3 for a test: STOP. Do not heal again.
• Report: "This test has been healed 3+ times. Recommend regenerating instead."
• Rationale: Over-healing compounds fragility; better to regenerate from the plan.

Healing Workflow:

1. Run test_run → identify failures
2. Call get_test_failures → get structured JSON (file, line, error, screenshot)
3. For each failure: run test_debug → pause at error
4. Use browser_snapshot + browser_generate_locator → inspect page state
5. Root cause analysis:
   • Stale selector? → check page object locator strategy first
   • Timing issue? → add Playwright auto-waiting (no manual waits)
   • Assertion mismatch? → STOP. Do not modify assertions. Mark test.fixme() with evidence, requires human approval
6. Apply fix based on root cause:
   • Locator issues: Update page object to use preferred strategy (getByRole > getByTestId), never CSS
   • Timing issues: Add proper awaits, never waitForTimeout
   • Assertion issues: Mark test.fixme(), document observed vs expected, flag for approval
7. Re-run specific test with test_run → verify
8. Call record_heal_event → log what was fixed with required fields:
   • healType: "selector" | "timing" | "logic" (use "assertion" ONLY with humanApprovedAssertionChange=true)
   • evidence: screenshot path or DOM description proving the locator was wrong, not the assertion
   • productBehaviorChanged: false for selector/timing fixes; true if app behaviour differs (triggers product bug flag)
9. Iterate until all pass

Key Rules:

• Fix one issue at a time, retest before moving on
• If test fails after 2 fix iterations and logic is correct: mark test.fixme() with explanation
• Always call record_heal_event after successful fix
• Do not ask user questions — do the most reasonable thing

Output: Patched .spec.ts file, heal event recorded in test-health.json

4. Orchestrator — playwright-test-orchestrator.agent.md

Role: Coordinate Planner → Generator → Executor → Healer with human review checkpoints.

Enforced Conventions:

• Inherits all conventions from Planner, Generator, and Healer
• Applies them sequentially with 3 human review checkpoints:

1. Checkpoint 1 (after Plan):

   • Present plan to user with scenario count and coverage
   • Ask: "Add, remove, or change any scenarios? Confirm Go."
   • Do not proceed until explicit confirmation

2. Checkpoint 2 (after Generate):

   • List all generated test files with scenario names
   • Ask: "Open any file to review. Should any be skipped? Confirm Go."
   • Do not proceed until explicit confirmation

3. Checkpoint 3 (after Execute, if failures):

   • Show failure table: test name, file, error, failed step
   • Ask: "Heal these automatically? Confirm Go or Skip."
   • Do not proceed unless explicitly confirmed

Key Workflow:

• Phase 1: Run Planner logic → save plan → checkpoint
• Phase 2: Run Generator logic for each scenario → checkpoint
• Phase 3: Run all tests → if all pass, jump to report; if fail → checkpoint
• Phase 4: Run Healer logic on each failure → re-run tests
• Phase 5: Generate final report with coverage & execution summary

Requirements Intake Modes:

• Jira issue key (for example, PROJ-123) via Jira MCP or Jira API
• Requirement files in requirements/
• Requirements pasted directly in chat

Incremental Generation Guard (mandatory):

• Scan existing generated tests for // req: REQ-NNN and // source: ...
• Skip scenarios already covered unless user explicitly requests regeneration
• If multiple requirement files exist, do not auto-generate; require explicit selection of one file per run

Rules Matrix — Which Agent Enforces What

How to Add a New Agent

If you need a new agent (e.g., API tester, component test planner):

1. Create .github/agents/<new-agent-name>.agent.md

   • Declare its role (what problem does it solve?)
   • List tools it needs (from playwright-test, playwright-qa-context, or custom servers)
   • Define its system prompt with clear workflow

2. Declare startup sequence

   • What tools must it call first (e.g., framework conventions)?
   • In what order?
   • Document as "non-negotiable"

3. Define rules it enforces (or inherits from parent agents)

   • What conventions must downstream agents respect?
   • How does this agent validate inputs from upstream?

4. Document integration points

   • What agent calls it (if any)?
   • What output does it produce?
   • Where is that output stored?

5. Update this file

   • Add a section under "Agent Roles & Responsibilities"
   • Update the Rules Matrix
   • Add to the Rules Audit section below

Rules Audit Checklist

Before merging a new agent or updating an existing one:

• Startup sequence is documented and non-negotiable
• Enforced conventions are listed explicitly (not implied)
• Each convention has a "why" (link to project pattern or rationale)
• Conflicts with existing agents are identified and resolved
• Integration with downstream agents is clear
• Output format is specified (file path, structure, required headers)
• Error handling is defined (what if the app doesn't have a fixture?)
• Guard rails are in place (e.g., heal limit, over-generation limit)

Conflict Detection

When rules might conflict:

1. Planner says "E2E tests use this flow" but app is missing that flow

   • Planner must flag as a gap, not generate a scenario for it
   • Orchestrator stops at Checkpoint 1 so user can decide

2. Generator creates a page object method that doesn't exist

   • Generator must add it to the page object file or fail gracefully
   • Rule: Page object extension is allowed; raw page.* in test is not

3. Healer finds a broken selector but test has been healed 3 times already

   • Healer must STOP and recommend regeneration
   • Do not override the heal limit for convenience

4. Generator's convention says "no hardcoded strings" but mock data doesn't have the required data

   • Generator must add it to shared/mock-data before using it
   • Better to expand mock data than to hardcode in test

Version Alignment

Agents are versioned by the Playwright tools they depend on:

• Playwright Test version → npx playwright run-test-mcp-server tools
• Custom MCP server version → playwright-qa-context tools

On upgrade:

1. Check Playwright release notes for new MCP tools
2. Update agent tools: lists if adopting new capabilities
3. Update system prompts if tool behavior changed
4. Test new agent against a sample app before committing

Record the Playwright version in each agent's comments:

# playwright-test-generator.agent.md
# Tested with: Playwright @1.45.0+
# Custom MCP: playwright-qa-context@1.0.0

References

• Startup sequences: each agent's "Startup sequence" section
• Rules matrix: see "Rules Matrix" table above
• Maintenance runbook: see CUSTOM-MCP.md for MCP server management
• Test validation: utils/validate-generated-tests.ts enforces rules at CI time