Detect Flaky Tests Without Masking Coding Agent Regressions

Last reviewed: 2026-09-15

Direct answer

Flaky test detection for coding-agent patches should treat a retry as a second observation, never as a replacement for the first. Run the affected test once, preserve its result and sanitized evidence, and then permit a small, bounded number of retries in a clean worker. Classify a first-attempt pass as passed, a fail-then-pass sequence as flaky, and a failure that exhausts its retries as failed. Widespread setup or service failures need a separate infrastructure-error classification.

The critical rule is that a recovered test must not silently become green. A pass on retry proves that the outcome changed; it does not prove that the patch is safe. Keep both attempts visible, compare the patch revision with its base revision under matched conditions, and require an explicit disposition for the flaky result. A strict merge policy can continue to fail a newly flaky check even when its retry passes.

Quarantine is a temporary containment mechanism, not a successful result. A quarantined test should still run in a scheduled lane, retain an owner and tracking issue, have an expiry date, and define the evidence required for release. Silent skips and open-ended quarantine gradually remove regression coverage.

Operator workflow: happy and error paths

1. Record the patch revision, base revision, selected tests, runner image, dependency state, concurrency setting, and random seed before execution. Keep the patch unchanged during classification.
2. Run each selected test once. Store a separate, sanitized attempt record before deciding whether to retry.
3. On the happy path, the first attempt passes. Mark the test passed, run the remaining required suites, and retain the result with the patch evidence. Do not spend retry budget on a passing test.
4. On the error path, the first attempt fails. Preserve its logs and artifacts, classify obvious setup-wide failures, and retry only the failed test in a fresh worker. Stop retrying if a configured failure cap indicates a broader outage.
5. If the retry passes, label the test flaky and keep the first failure visible. Run the same test at the base revision in an equivalent environment. A single flip cannot establish whether the patch created the instability, exposed an existing problem, or encountered infrastructure noise.
6. If every bounded attempt fails, keep the merge blocked and investigate it as a repeatable failure. Reproduce the smallest affected test before asking the coding agent to modify production or test code.
7. If temporary quarantine is necessary, create a record with an owner, reason, first-seen revision, tracking issue, expiry, scheduled execution lane, and exit criteria. Remove the quarantine only after the repair satisfies a team-defined clean-run threshold.

This workflow separates two questions that are often mixed together: whether the result is stable and whether the patch caused the failure. Retries help answer the first question. Matched base-versus-head runs, failure artifacts, and code review help answer the second.

Who this is for

This guide is for engineers who review pull requests produced or modified by coding agents, especially teams with unit, integration, or browser suites in CI. It also applies to build-infrastructure maintainers who need to introduce retries without weakening merge protection.

The workflow assumes that a human remains responsible for merge policy and quarantine decisions. A coding agent can reproduce failures, inspect state leakage, or propose a repair, but it should not erase the first failure, loosen an assertion, or add an indefinite skip merely to obtain a passing check.

Key takeaways

• Preserve every attempt. The first failure is evidence even when a later attempt passes.
• Use bounded, targeted retries in clean workers. Re-running an entire pipeline obscures which test changed outcome.
• Give fail-then-pass its own flaky classification instead of converting it to passed.
• Compare base and patch revisions under equivalent conditions before attributing instability to the patch.
• Cap retries when many tests fail so an infrastructure outage cannot create a retry storm.
• Make quarantine visible, owned, scheduled, and time-limited.
• Require the repair to address the root cause rather than merely weakening the test.

Sources checked

The pytest flaky-test guidance defines flaky behavior as intermittent failure and identifies uncontrolled system state, insufficient isolation, parallel execution, test-order dependence, strict timing assertions, and thread interactions as potential causes. It also describes non-strict expected-failure marking as a form of manual quarantine that is dangerous when left in place permanently.

The Playwright retry documentation shows a useful outcome model: passed on the first run, flaky after a failed run followed by a successful retry, and failed after all retries. It also documents replacing a failed worker process before continuing, which prevents a failed test from contaminating later tests in that worker.

The Gradle Test Retry Plugin documents bounded retries, a cap on total failures before retrying stops, discrete reporting for repeated executions, and a setting that can fail a task when a failed test later passes. Its documentation explicitly cautions that retries alone are not a complete mitigation strategy and must be paired with tracking and repair.

Together, these sources support a provider-neutral contract: isolate attempts, preserve outcome transitions, bound retry work, and keep discovered instability actionable.

Contract details to verify

A reliable implementation needs explicit contracts for test identity, retry eligibility, outcome aggregation, evidence retention, and quarantine. Verify those contracts before enabling retries across the repository.

Outcome contract

Do not infer that every repeated failure is a product regression. A missing service, exhausted disk, broken fixture setup, or unavailable dependency can make many unrelated tests fail together. Conversely, do not infer that fail-then-pass is harmless. It is evidence of nondeterminism somewhere in the test, product, or environment.

Retry and quarantine policy

The following is an illustrative policy contract, not a universal set of thresholds:

retry_policy:
  enabled_in_ci: true
  max_retries: 1
  max_failures_before_abort: 20
  require_fresh_worker: true
  passed_after_retry: fail_and_mark_flaky
quarantine_policy:
  owner_required: true
  issue_required: true
  expires_after_days: 7
  scheduled_lane_required: true
  exit_criteria_required: true

Choose the numeric limits from suite size, runtime, and failure history. The important properties are bounded work, a clean retry boundary, a distinct flaky outcome, and an escape hatch for broad infrastructure failures. Where a runner retries a class or serial group rather than one test, record that granularity because passing neighbors may be executed again too.

Sanitized per-attempt logging

Emit one allowlisted event per attempt. Use stable test identifiers and short revision identifiers, and store large artifacts separately. A minimal event can look like this:

schema_version: test-attempt.v1
run_id: ci-4821
revision: abc1234
test_id: checkout_refund
suite: integration
attempt: 1
outcome: failed
duration_ms: 1820
worker: linux-2
seed: 42
failure_class: assertion
artifact_ref: attempt-1
sanitization: allowlist

Useful fields include the base revision, patch revision, test ID, suite, attempt number, worker class, seed, duration, outcome, normalized failure class, artifact reference, and quarantine ID when applicable. An aggregate record may summarize the sequence, but it must not overwrite the attempt records.

Sanitize before upload. Do not collect the raw process environment, request headers, browser storage, database rows, fixture payloads, or full command lines by default. Review stack traces, screenshots, traces, and captured output for sensitive or user-provided data before retaining them. Record that sanitization occurred so reviewers know an artifact was deliberately filtered rather than accidentally truncated.

Attribution contract

Run the smallest failing selection at both the patch revision and its base using the same runner class, dependency state, seed, parallelism, and fixture setup. Head-only repeatable failure is strong evidence associated with the patch. Mixed outcomes at both revisions suggest an existing flaky test or shared infrastructure problem. A head-only fail-then-pass result deserves investigation but is not, by itself, proof that the patch introduced the flake.

If parallel execution changes the result, follow the dedicated test-order dependency checks . If timing or shared-state contention is suspected, use reproducible race-condition testing to capture schedules and repeated evidence.

Quarantine contract

A quarantine entry should contain a stable test ID, owner, tracking issue, reason, first-seen revision, creation date, expiry date, scheduled execution lane, latest result, and exit criteria. The scheduled lane must retain every attempt and alert on changes. Expiry should trigger review or escalation, not automatic renewal. If the test covers unique behavior, document the temporary coverage gap and prioritize repair accordingly.

Failure modes

A retry erases the original failure

Some reporting pipelines keep only the final attempt. The pull request appears green, reviewers cannot see the outcome transition, and the flake becomes indistinguishable from a clean pass. Fix this by writing immutable attempt records first and deriving the aggregate classification afterward.

The whole workflow is rerun

A full workflow rerun changes too many variables: worker allocation, test order, dependency availability, cache state, and elapsed time. It may produce a green check without identifying which test flipped. Retry the smallest supported test unit and retain the original workflow as evidence.

The retry inherits contaminated state

A retry in the same process can reuse global variables, files, browser state, ports, threads, or databases left by the failure. A pass or second failure then measures a different condition. Prefer a fresh worker and recreate fixtures from declared inputs. If a runner must retry a serial group, report the entire group as the retry unit.

A broad outage triggers a retry storm

When a database, test service, disk, or setup phase fails, hundreds of retries add load while producing little diagnostic value. Count failures per round and stop retries after a configured cap. Label the run as an infrastructure error so the patch is held without falsely attributing every failure to its code.

Quarantine becomes permanent

An unowned skip can survive for months while coverage silently shrinks. Require an owner, issue, expiry, scheduled run, and exit criteria. Report the quarantine count and oldest entry. Do not let the coding agent extend an expiry or remove a scheduled run without explicit review.

The test is weakened instead of repaired

A coding agent may make the check pass by increasing a timeout, broadening a tolerance, deleting an assertion, or adding a skip. Those changes can conceal a product defect. Review the behavioral contract before accepting a test-only repair, and require evidence that the new assertion still detects the original failure condition.

Base and patch runs are not comparable

Changing dependencies, worker images, seeds, or concurrency between revisions makes attribution weak. Capture comparison inputs before execution and identify every unavoidable difference. If clean reproduction remains impossible, retain an unknown classification rather than declaring either the patch or the test innocent.

FAQ

Is a test successful if it passes on retry?

It completed successfully on one attempt, but the overall observation is flaky because its outcomes changed. Keep that classification visible. A team may define a reviewed exception for known instability, but silently reporting passed destroys the signal needed to fix it.

How many retries should CI allow?

There is no universal number. Start with the smallest useful bound, often one retry for classification, and cap total failures so widespread outages stop quickly. More attempts increase cost and can make an unstable test appear healthy by chance. Base the final threshold on measured suite behavior.

When is quarantine appropriate?

Use it when a confirmed unstable test is blocking unrelated work and cannot be repaired immediately. Quarantine only after preserving evidence, assigning an owner, scheduling continued execution, setting an expiry, and documenting the coverage impact. It should not be the automatic response to every first failure.

How do I distinguish a flaky test from flaky infrastructure?

Look at scope and reproducibility. One test that flips under matched conditions is a flaky-test candidate. Many unrelated tests failing during setup point toward infrastructure. Compare base and patch revisions, isolate workers, retain seeds and timings, and avoid forcing certainty when the evidence supports only an unknown classification.

Should the coding agent repair the failure automatically?

It can propose a focused repair after classification and reproduction. Keep merge policy, assertion changes, and quarantine decisions under human review. The repair should explain the root cause, preserve the intended behavior, and include evidence from both the failure and the corrected run.

Reader next step

Choose one frequently used CI suite and implement the four outcome classes: passed, flaky, failed, and infrastructure error. Add one clean-worker retry, fail recovered tests during the trial, emit the allowlisted per-attempt fields, and set a total-failure cap. Exercise the reporter with a disposable test fixture that fails once and then passes, confirming that both attempts remain visible.

Then define the quarantine record and expiry behavior before allowing any test to leave the blocking lane. Integrate the resulting evidence into your coding-agent CI repair workflow , and require reviewers to resolve every flaky classification explicitly.