Last reviewed: 2026-09-13

Direct answer

A coding agent changed-line coverage gate should evaluate the final pull-request diff, not merely the repository-wide coverage percentage. Define the comparison base, run the relevant test suites with branch measurement enabled, generate a fresh machine-readable report, and compare that report with the changed executable lines. Fail the job when tests fail, the base cannot be resolved, the report is absent or invalid, or the changed-line percentage falls below the declared threshold.

The diff-cover project defines diff coverage as the percentage of new or modified lines covered by tests. Its command can compare an XML or LCov report with a Git diff and return a nonzero status below a configured threshold. That makes the metric suitable for a required CI check, provided the surrounding workflow treats missing evidence as an error.

Changed-line coverage is narrow evidence. It shows that execution reached modified statements; it does not prove that assertions are meaningful or that behavior is correct. Branch measurement adds an important second signal. The Coverage.py branch measurement guide explains that a conditional line can execute while one possible destination remains unvisited. Reviewers should therefore inspect modified decisions for missing paths even when line coverage passes.

Who this is for

This workflow is for maintainers, test engineers, and CI platform teams that review patches produced with coding agents. It is especially useful in a mature repository where a project-wide percentage can hide an untested change inside a large denominator.

The gate also helps reviewers separate two questions: whether existing coverage moved across the whole project, and whether the code changed in this pull request was exercised. It is not a substitute for behavioral review, security checks, integration tests, or human judgment. Treat it as one bounded merge condition in a larger verification system.

Key takeaways

  • Measure the pull-request patch separately from overall project coverage.
  • Resolve and record both base and head revisions so the same diff can be reconstructed.
  • Generate coverage from a clean run of the relevant suites; never reuse an unexplained report.
  • Fail closed when tests, report generation, path mapping, or comparison-base resolution fail.
  • Measure branches as well as statements, then inspect changed conditionals with missing destinations.
  • Keep the threshold, exclusions, report format, and status-check name in reviewed configuration.
  • Preserve a small sanitized result record plus detailed CI artifacts.
  • Do not interpret a passing coverage score as proof that a test can detect a defect.

Sources checked

  • diff-cover documents changed-line comparison, supported report inputs, explicit comparison branches, uncovered-line output, JSON reports, and nonzero exit status through --fail-under.
  • Codecov status checks distinguish project status from patch status and document configurable targets, thresholds, path or flag filters, missing-report behavior, and pull-request gating.
  • Coverage.py branch coverage measurement explains statement opportunities, branch destinations, partial branches, and branch data in XML and JSON reports.
  • GitHub protected branch documentation confirms that status checks can be required before merge and warns that duplicate job names across workflows can make required-check results ambiguous.

Contract details to verify

Write the gate as an explicit contract that a reviewer can reconstruct. A useful contract records these details:

  1. Diff identity. Name the intended pull-request base ref and resolve it to a revision. Record the tested head revision as well. If the base changes after a rebase or target-branch update, regenerate the evidence.
  2. Test scope. Declare which unit, integration, and component suites contribute coverage. An agent-created test alone is not enough if the modified code is exercised through other suites or services.
  3. Report provenance. Erase old measurement data before the run, combine every expected shard, and generate one supported report from that run. A report left by an earlier commit must never satisfy the gate.
  4. Metric definition. Count executable added or modified lines as the changed-line denominator and covered changed lines as the numerator. Record branch measurement separately so a line hit does not conceal an untested destination.
  5. Policy result. Set a documented threshold and a stable failure policy. Decide deliberately how generated files, migrations, and structurally partial branches are handled. Any exclusion should be visible in review.
  6. Evidence output. Retain the resolved revisions, suite identity, report format, counts, percentage, threshold, result, and artifact path. Keep detailed missing-line information in an artifact rather than flooding the main log.
  7. Merge enforcement. Publish one uniquely named check for the tested head revision and require it on the protected branch. Do not reuse the same job name for unrelated workflows.

Connect the test scope to the patch before running the gate. A short change-scope note gives the reviewer a concrete list of affected modules, interfaces, and expected tests.

Happy path operator workflow

Start from a clean checkout of the exact pull-request head. Ensure the intended base ref is present, run the selected suites under branch measurement, generate XML, and compare it with that base. The following Python-oriented example uses the documented Coverage.py and diff-cover interfaces:

set -euo pipefail

BASE_REF="${BASE_REF:-origin/main}"
REPORT_DIR="${REPORT_DIR:-artifacts/coverage}"

mkdir -p "$REPORT_DIR"
git rev-parse --verify "$BASE_REF^{commit}" >/dev/null
coverage erase
coverage run --branch -m pytest
coverage xml -o "$REPORT_DIR/coverage.xml"
diff-cover "$REPORT_DIR/coverage.xml" --compare-branch="$BASE_REF" --fail-under=90 --show-uncovered --format "json:$REPORT_DIR/changed-coverage.json"
printf '%s\n' 'coverage_gate=pass'

The 90 percent value is illustrative, not a universal recommendation. Choose a repository policy based on risk, existing test architecture, and reviewed exceptions. Pin the measurement tools in the repository’s normal dependency mechanism so local reproduction and CI use compatible behavior.

On success, upload the XML and changed-coverage JSON as artifacts, emit the sanitized summary, and publish the uniquely named required status. A reviewer should then inspect uncovered changed lines, modified conditionals, test assertions, and any coverage configuration changed by the same patch.

Use an allowlisted log event rather than dumping the process environment, source text, test inputs, or absolute workspace paths:

{
  "event": "changed_line_coverage_gate",
  "repository": "service-a",
  "base_ref": "origin/main",
  "base_revision": "1a2b3c4",
  "head_revision": "8f3c2d1",
  "suite": "unit",
  "report_format": "coverage-xml",
  "changed_executable_lines": 18,
  "covered_changed_lines": 17,
  "changed_line_percent": 94.44,
  "partial_branches": 1,
  "threshold_percent": 90,
  "result": "pass",
  "failure_code": null,
  "artifact_path": "artifacts/coverage/changed-coverage.json"
}

Error path operator workflow

Handle failures by evidence class instead of retrying everything blindly:

  • If the test command exits nonzero, stop with tests_failed. Preserve the test report, but do not calculate a passing patch score from partial coverage.
  • If the base ref cannot be resolved, stop with comparison_base_unavailable. Restore the required history or correct the target ref; do not silently compare against a convenient local branch.
  • If XML generation fails, the file is empty, or paths cannot be mapped to the checkout, stop with coverage_report_invalid.
  • If the changed-line percentage is below policy, return below_threshold, retain the uncovered-line artifact, and add behavior-focused tests or reduce unrelated changes in the patch.
  • If an expected suite or shard did not report, return coverage_input_incomplete. Do not treat the available subset as the full denominator.
  • If CI infrastructure fails, report an error rather than a neutral success. Rerun against the same head, then regenerate all evidence if the head changes.

A successful rerun should replace the failed summary only for the exact same head revision. A new commit starts a new evidence cycle.

Failure modes

The project percentage masks the patch. A large, well-covered codebase may barely move when several new lines are untested. Keep project and patch statuses separate; the Codecov documentation explicitly describes them as different measurements.

The comparison base is wrong. Comparing against a stale local branch can omit lines or include unrelated work. Log the resolved base revision, not only a symbolic name.

Coverage data is stale or mixed. Reusing files from a previous run, combining different heads, or losing a shard breaks provenance. Erase old data and make completeness an explicit condition.

Statement coverage hides an untested decision. A test can execute an if line while never taking its error path. Branch measurement exposes missing destinations, but the reviewer still has to decide which paths matter.

The patch edits its own measuring rules. Changes to exclusions, pragmas, suite selection, thresholds, or report paths can make a score easier to pass. Route those configuration changes for explicit review and require a clean run under the accepted policy.

Tests execute code without checking behavior. Assertions may be absent, tautological, or aimed at implementation details. Coverage only records execution. For stronger evidence, test fault-detection strength with mutation testing .

Renames and generated sources map poorly. A report path that does not match the checked-out source can produce missing or misleading results. Treat mapping warnings as gate errors until the report roots and generated-file policy are verified.

No-report behavior is permissive. Some status systems can be configured to pass or omit a check when coverage is absent. A required gate should fail when its expected report is not found.

The required check is ambiguous. GitHub warns that identical job names across workflows can block merging through ambiguous status results. Give the coverage job a unique, stable name and constrain which integration may publish it where supported.

A flaky test produces unstable evidence. Do not raise the coverage score by accepting an unexplained retry. Diagnose the test, retain the first failure, and regenerate the complete report after the suite is stable.

FAQ

Should changed-line coverage be 100 percent?

Not automatically. Full coverage of modified executable lines is a clear policy, but generated adapters, defensive branches, and language-specific measurement behavior may need reviewed handling. A lower threshold can also be reasonable during adoption. Whatever value you choose, keep it versioned, visible, and consistent across contributors.

Should project coverage also be required?

Often, yes. Patch coverage asks whether modified lines ran; project coverage detects broader movement in the repository. Requiring both prevents an apparently strong patch from accompanying a large unmeasured change elsewhere, while preventing old coverage debt from hiding new untested code.

Does branch measurement replace changed-line coverage?

No. They answer related but different questions. Changed-line coverage focuses the review on the diff. Branch measurement identifies possible control-flow destinations that were not taken. Use both, then inspect changed conditions whose alternate paths remain missing.

What should happen when a pull request changes only comments or deletions?

The contract should define the no-executable-lines result explicitly. A neutral or successful result can be appropriate after the tool confirms that the denominator is genuinely empty. Missing reports and failed path mapping are different conditions and should remain errors.

Can the coding agent decide which tests count?

It can propose a test scope, but repository-owned CI configuration should make the final selection. Reviewers should compare the proposed scope with module boundaries, callers, integrations, and the change-scope note. The gate should be recomputed by CI rather than accepting a self-reported percentage in the patch description.

What should a reviewer inspect after the gate passes?

Read the tests, verify meaningful assertions, inspect error and boundary behavior, check modified coverage configuration, and confirm that the status belongs to the latest head. Coverage answers whether execution arrived; review determines whether the test established the intended behavior.

Reader next step

Add the gate in two controlled stages. First, run it as an informational job for several representative pull requests and record mapping errors, empty-diff behavior, typical scores, and suite completeness. Then commit the threshold and failure contract to reviewed CI configuration.

Before making the check required, create one small fixture patch with an uncovered executable line and confirm that the job fails. Add a behavior-focused test and confirm that the same job passes. Repeat with a two-way conditional to ensure branch measurement reveals the unvisited destination. Finally, assign a unique status-check name, require it on the protected branch, and retain the XML, diff report, and sanitized summary for each head revision.

If the first required run fails, follow a bounded CI repair loop : classify the failure, change one cause, rerun the complete evidence path, and review the final diff before merge.