Last reviewed: August 12, 2026.

Direct answer

Treat every coding agent UI patch as a set of user-visible states, not merely as a file diff. Before merge, run an automated accessibility scan against each changed state, inspect inconclusive findings, and complete a focused human review of keyboard operation, focus behavior, names, instructions, and status changes. A green automated scan is useful evidence, but it is not proof that the interface is accessible.

That distinction is explicit in the Playwright accessibility testing guide : automated tests can detect common problems, while many problems require manual assessment. The W3C evaluation overview likewise says that no tool alone can determine whether a site meets accessibility standards and that knowledgeable human evaluation is required. Deque reports in the axe-core project documentation that axe-core finds an average of 57% of WCAG issues automatically and returns incomplete results when it cannot make a certain determination.

A sound merge gate therefore has two independently required parts: repeatable automation for detectable issues and a documented human check for behavior that automation cannot settle. If either part fails to run, the result is inconclusive rather than passed.

Who this is for

This workflow is for frontend engineers, test engineers, platform teams, and pull request reviewers who allow coding agents to change pages, components, styles, forms, navigation, or client-side behavior. It is especially useful when an agent can produce a visually convincing patch faster than a reviewer can inspect every rendered state.

The approach assumes that the repository can render the affected interface in a browser test environment. It does not require every reviewer to be an accessibility specialist. It does require an owner who can define the states that matter, recognize when a finding needs expert review, and prevent the test runner from failing open.

Key takeaways

  • Inventory changed routes, components, and transient states before choosing what to scan.
  • Run the scan only after the interface reaches the intended state; hidden menus and unopened dialogs are otherwise easy to miss.
  • Treat violations, incomplete findings, manual failures, and test infrastructure errors as different outcomes.
  • Do not use broad exclusions or disabled rules as permanent substitutes for fixing defects.
  • Record compact, sanitized evidence that a reviewer can compare without collecting page contents or user-entered values.
  • Require automation and human review to pass before the accessibility gate is green.

Sources checked

  • The Playwright accessibility testing guide documents whole-page and scoped axe scans, state setup before analysis, WCAG-tagged rules, known-issue handling, and test attachments. It also warns that automated testing cannot find every accessibility problem.
  • The W3C Evaluating Web Accessibility Overview recommends evaluating early and throughout development, explains the role of tools, and requires knowledgeable human evaluation rather than tool-only conclusions.
  • The Deque axe-core repository describes its supported rule families, integration with functional tests, automatic findings, and incomplete results that need manual review.

These sources support the testing model and its limits. The merge policy, evidence fields, and workflow below are an operational implementation of that model.

Contract details to verify

Start by linking the pull request to a change scope note . The note should identify what the agent changed and, just as importantly, what it was not asked to change. Then make the accessibility contract explicit in the pull request.

Define the state matrix

List each affected surface and the states a user can reach. A profile form might require default, populated, validation-error, saving, success, and server-error states. A navigation change might require collapsed, expanded, keyboard-focused, and small-viewport states. A dialog must be opened before it can be meaningfully scanned.

For each state, record:

  • The route or component under test.
  • The interaction needed to expose the state.
  • The element that proves the state is ready.
  • Whether the scan covers the full page or a deliberately narrow region.
  • The automated rule configuration.
  • The manual checks required for that interaction.

Playwright documents that AxeBuilder.analyze() scans the page in its current state. Its example opens a navigation menu, waits for the flyout, and only then analyzes it. Use the same principle for tabs, drawers, validation messages, autocomplete results, loading transitions, and content revealed after a request.

Set unambiguous outcomes

Use three outcomes instead of reducing every run to green or red:

  • passed: the expected state was reached, the configured scan completed, no disallowed violations appeared, incomplete findings were resolved, and the human checklist passed.
  • failed: an automated violation or manual accessibility defect is attributable to the candidate patch, or a required exception has expired.
  • inconclusive: the page did not reach the expected state, the browser or scanner crashed, evidence is missing, or a finding needs knowledgeable evaluation before it can be classified.

An inconclusive run must not satisfy a required check. Retry it under the same pinned configuration. If it remains inconclusive, route it to a human owner instead of teaching the coding agent to suppress the symptom.

Happy-path operator workflow

  1. Read the patch scope and map changed markup, styling, events, and state transitions to the state matrix.
  2. Build the base commit and candidate commit with the same locked dependencies, browser project, viewport, test data, and scanner configuration.
  3. Run the existing functional test for each state. Confirm the expected state marker is visible before starting the accessibility scan.
  4. Run a full-page scan for each changed route. Add a scoped scan when a transient component needs a precise assertion, but do not let that narrow scan replace coverage of the surrounding page.
  5. Review both violations and incomplete results. Deque documents incomplete results specifically as cases that need manual review.
  6. Complete the human checklist: traverse the workflow by keyboard, check that focus moves and remains visible, confirm controls have understandable names and instructions, and verify that errors or status changes are perceivable in the interaction.
  7. Attach sanitized findings and the manual checklist result to the pull request. Have a reviewer confirm that the state matrix matches the actual diff.
  8. Mark the gate passed only when all required states have completed both parts of the contract.

This workflow complements a visual regression workflow . Screenshot comparison can reveal appearance changes, while the accessibility gate examines semantics and interaction behavior that a screenshot alone cannot establish.

Log evidence without collecting page contents

Keep the record small enough to review and safe enough to retain. A useful event can look like this:

{
  "run_id": "a11y-042",
  "commit_ref": "abc1234",
  "route": "/profile",
  "ui_state": "error-shown",
  "browser": "chromium",
  "scan_scope": "full-page",
  "tool_version": "pinned",
  "rule_id": "label",
  "impact": "serious",
  "target": "#email",
  "result": "failed",
  "manual_checks": ["keyboard", "focus", "name"],
  "artifact_ref": "ci-184"
}

Log identifiers, classifications, counts, and artifact references. Do not log raw rendered HTML, form values, user content, cookies, request headers, or full page dumps by default. If a selector itself exposes user data, replace it with a stable test identifier before retaining the event. Store detailed scan attachments behind the same access and retention controls used for other CI artifacts.

Error-path operator workflow

  1. Preserve the failing state name, rule identifier, affected selector, scanner configuration, and manual observation.
  2. Reproduce the same state on the base commit and candidate commit. A base failure is pre-existing evidence; it does not automatically excuse a new or expanded failure in the candidate.
  3. If the scanner reports incomplete, assign a knowledgeable reviewer to decide whether the element passes, fails, or requires a different test. Do not silently convert incomplete to passed.
  4. Give the coding agent the smallest reproducible state and the expected behavior. Ask for a focused fix rather than a broad accessibility cleanup unrelated to the patch.
  5. Rerun the complete affected state matrix after the fix. A change that repairs one label can still alter focus, visibility, or another state.
  6. If policy allows a temporary exception, record the exact rule and target, an owner, a reason, and an expiry condition. Keep the exception narrower than the component and fail the check when it expires.

Playwright warns that excluding an element also excludes its descendants and prevents all rules from running on that element. It also describes rule disabling as a temporary response to known issues. Prefer a small, reviewable fingerprint of an accepted finding over a snapshot of an entire result object, which its guidance describes as fragile.

Failure modes

Scanning only the initial page state. The default view passes while the agent-generated menu, dialog, error summary, or loading result is never tested. Fix this by making state setup and readiness assertions part of the contract.

Running the scanner before the UI settles. The test analyzes an empty container or stale state. Wait for a state-specific element, not an arbitrary delay, before scanning.

Treating zero violations as certification. Automated tools cover only automatically decidable issues. The cited Playwright, W3C, and Deque guidance all preserve a role for human evaluation.

Failing open on infrastructure errors. A crashed browser, missing report, navigation timeout, or scanner exception is reported as zero findings. Model these conditions as inconclusive and keep the required check non-green.

Hiding debt with broad exclusions. Excluding a container can remove all of its descendants from every rule. Disabling a rule globally can hide a new defect on an unrelated surface. Require the smallest possible exception, ownership, and expiry.

Comparing against a stale baseline. A baseline produced with a different browser, state, dependency set, or rule configuration creates noisy or misleading deltas. Generate base and candidate evidence under the same configuration.

Ignoring incomplete findings. Axe-core uses incomplete results for cases it cannot decide. Dropping that collection discards the queue for human judgment.

Logging excessive evidence. Full DOM fragments and page dumps can contain information unrelated to the defect. Retain rule, target, state, classification, and a controlled artifact reference instead.

Letting the agent optimize for the check. A patch can remove a failing element, disable a rule, or narrow the scan without improving the user experience. Review configuration changes and test deletions as carefully as production code.

FAQ

Is an axe scan enough for a small patch?

No. It is still valuable, but patch size does not determine whether a problem is automatically detectable. A one-line event or markup change can affect focus, naming, or status behavior. Use a shorter human checklist for a small surface, but do not remove the human part entirely.

Should CI block every pre-existing violation?

Not necessarily on the first rollout. A practical adoption path records a reproducible baseline and blocks new or expanded findings while existing debt is assigned. The baseline must be specific enough that a new failure cannot hide behind an old rule-level exception. Manual failures introduced by the candidate should still block the patch.

Should the scan cover a component or the whole page?

Use both when appropriate. A component scan is useful for a transient state and faster diagnosis. A full-page scan catches interactions between the component and its context. Playwright supports scoped scans, but its warning about exclusions illustrates why narrow scope should be deliberate.

How often should the human checklist run?

Run it for every changed interaction covered by the gate. Broader evaluation and inclusive user testing can follow the product’s risk and release cadence, but they do not justify skipping review of the behavior changed in the current patch.

What belongs in a temporary exception?

Record the exact finding, target, affected state, owner, reason, and removal condition. Avoid copying complete page content into the exception. Review the exception whenever the component or scanner configuration changes.

How is this different from ordinary functional testing?

Functional tests usually establish that an action completes. The accessibility contract also checks whether a user can find, understand, and operate that action through the required interaction modes. The two test types should share state setup where possible, but they answer different questions.

Reader next step

Choose one recent coding agent UI pull request and write a state matrix for its changed surface. Add one automated scan after an explicit readiness assertion, run the keyboard and focus checklist, and attach the sanitized event fields above. Intentionally introduce a missing label once to verify that CI reaches the error path and cannot fail open, then remove the defect and rerun the full matrix.

When a reviewer finds a problem, turn the evidence into a focused coding agent follow-up with one reproducible state, one expected behavior, and the requirement to rerun every affected state before merge.