Last reviewed: September 12, 2026

Direct answer

Reliable coding agent performance regression testing compares a candidate patch with a known baseline under the same declared conditions. A useful gate verifies correctness first, warms up the measured code when necessary, collects repeated measurements, stores machine-readable results, and evaluates them against thresholds chosen before the patch runs. It does not fail or pass a patch from one stopwatch value.

The gate needs two distinct boundaries. A noise threshold describes changes too small to interpret confidently in the chosen environment. A regression limit states how much repeatable degradation the project is willing to accept. A result can therefore be outside the noise band yet still remain inside the permitted budget. Keeping those concepts separate prevents statistical sensitivity from becoming an accidental product requirement.

When a result exceeds the regression limit, rerun it according to a fixed error-path procedure on the same testbed. If the slowdown repeats, block the patch and give the coding agent the baseline artifact, candidate artifact, benchmark contract, correctness result, and relevant diff. If the result does not repeat, classify the run as inconclusive or environmentally invalid. Do not keep retrying until a favorable sample appears.

Who this is for

This guide is for developers and reviewers handling performance-sensitive changes produced by coding agents: parsers, serializers, database access layers, build tools, allocation-heavy routines, and frequently called library code. It assumes the repository already has a correctness test for the behavior being measured.

It is also useful to CI operators defining a reproducible testbed. Benchmark infrastructure is part of the review surface. If a patch changes both the measured implementation and the benchmark contract, reviewers should examine those changes independently instead of accepting a newly weakened gate as evidence of improvement.

Key takeaways

  • Compare a target-branch baseline and candidate on the same testbed, build mode, input, and metric.
  • Run correctness assertions before interpreting speed. Fast incorrect output is a failure.
  • Declare warm-up, repetitions, statistic, noise threshold, and regression limit in version-controlled policy.
  • Preserve complete machine-readable results rather than copying only a console summary.
  • Treat missing baselines, incompatible environments, excessive noise, and malformed results as errors, not passes.
  • Give a coding agent a narrow evidence packet only after the operator can reproduce the slowdown.

Sources checked

  • The Google Benchmark user guide documents discarded warm-up periods, repeated runs, random interleaving, aggregate statistics, dry runs, and JSON output containing benchmark and execution context.
  • The Criterion.rs command-line output guide explains confidence intervals, saved-baseline comparisons, configurable noise thresholds, outlier reporting, and the effect of inconsistent machine conditions.
  • The pytest-benchmark usage guide documents calibration, minimum rounds, precision controls, warm-up, saved runs, JSON reports, baseline comparison, and expressions that fail a test on excessive regression.

Contract details to verify

Define the measured behavior

Name one behavior and one representative input for each benchmark. The name should remain stable across the baseline and candidate. Record whether lower or higher values are better and choose a unit that matches the behavior, such as CPU time per parse or items processed per second.

Keep fixture creation, file discovery, and unrelated setup outside the timed region unless those operations are deliberately part of the performance contract. Run a correctness assertion against the result. If the input or expected result changes, create a new benchmark identity or update the baseline through explicit review.

A compact policy can look like this:

schema_version: 1
benchmark: parse_manifest/large
metric: cpu_time_ns
direction: lower_is_better
minimum_repetitions: 10
warmup_seconds: 1
noise_threshold_percent: 3
regression_limit_percent: 5
baseline: target_branch_same_testbed

These numbers are an example, not universal defaults. Calibrate them from the variance and performance requirements of the repository. The important property is that the policy exists before a candidate result is evaluated.

Pin the comparison conditions

Build baseline and candidate with the same compiler, optimization level, dependencies, benchmark version, input data, and runner class. Run them close enough together that the machine configuration is comparable. Record the relevant configuration in the result artifact.

Machine state matters. The Criterion.rs documentation gives the concrete warning that merely changing a laptop between battery and wall power can produce apparent improvements or regressions. Shared-runner load, thermal changes, and background processes can likewise make a comparison noisy. A canonical gate should therefore use a controlled runner and reject runs whose required context differs from the baseline.

For parallel repository work, isolate the baseline and candidate in separate Git worktrees so building one revision does not contaminate the other.

Warm up, repeat, and retain evidence

Warm-up is appropriate when caches or runtime initialization would otherwise dominate early measurements. Google Benchmark can discard results collected during a minimum warm-up period. Criterion.rs also performs a warm-up before sample collection. The contract should state whether warm-up is required and how long it lasts.

Repeated runs expose variability that a single value hides. Google Benchmark reports aggregate statistics when repetitions exceed one and can randomly interleave repetitions to reduce the effect of changing system state. Preserve the individual results whenever possible. Aggregates help a reviewer read the result, but raw observations are needed to investigate outliers or a changed distribution.

Use a dry run to catch missing fixtures and broken benchmark binaries before spending time on full measurement:

./candidate/build/benchmarks --benchmark_filter='BM_ParseManifest.*' --benchmark_dry_run

Then run the baseline and candidate with matching controls:

./baseline/build/benchmarks --benchmark_filter='BM_ParseManifest.*' --benchmark_min_warmup_time=1 --benchmark_repetitions=10 --benchmark_enable_random_interleaving --benchmark_out=artifacts/baseline.json --benchmark_out_format=json
./candidate/build/benchmarks --benchmark_filter='BM_ParseManifest.*' --benchmark_min_warmup_time=1 --benchmark_repetitions=10 --benchmark_enable_random_interleaving --benchmark_out=artifacts/candidate.json --benchmark_out_format=json

Projects using Python can apply the same contract with saved pytest-benchmark runs, minimum rounds, optional precision targets, JSON output, comparison, and a declared failure expression. The syntax differs, but the review questions are the same.

Happy-path operator workflow

  1. Verify that the benchmark policy, input fixture, and expected result match the target branch.
  2. Run the correctness suite for the affected behavior. Stop if it fails.
  3. Build baseline and candidate revisions in clean, isolated directories with identical release settings.
  4. Run the benchmark dry check, followed by the complete baseline and candidate commands.
  5. Validate both result files. Reject missing benchmarks, nonnumeric metrics, incompatible units, unequal testbeds, or fewer repetitions than the policy requires.
  6. Compute the selected comparison consistently. Do not compare a baseline median with a candidate mean.
  7. If the result remains within the permitted regression limit and the run satisfies the noise checks, mark the performance gate as passed.
  8. Attach the policy, baseline result, candidate result, correctness status, and sanitized decision record to review.

A pass means the candidate met this declared benchmark contract on this testbed. It is not proof that every workload or production environment became faster.

Error-path operator workflow

If the candidate crosses the regression limit, stop the merge path. Confirm that both artifacts refer to the expected revisions and that correctness still passes. Perform the one verification rerun allowed by policy on the same quiet testbed, replacing neither original artifact.

If the slowdown repeats, mark the gate failed. Give the coding agent the benchmark name, direction, observed range, allowed limit, baseline and candidate artifacts, and files changed. Ask for a focused diagnosis rather than an unspecified optimization. A good request should forbid weakening the benchmark, deleting assertions, or changing the threshold without separate approval.

If the slowdown does not repeat and the runs show high dispersion or many outliers, mark the comparison inconclusive. Investigate the runner or benchmark design, then collect a fresh pair. If the baseline is missing, stale, built differently, or incompatible with the candidate schema, report a configuration error. Never convert an invalid comparison into a pass.

Use sanitized logging fields such as these:

{
  "event": "benchmark_gate",
  "run_id": "perf-pr-1842",
  "candidate_commit": "a1b2c3d",
  "baseline_commit": "d4e5f6a",
  "benchmark": "parse_manifest/large",
  "testbed": "linux-x86_64-pinned",
  "metric": "cpu_time_ns",
  "baseline_median": 29275,
  "candidate_median": 32100,
  "change_percent": 9.65,
  "noise_threshold_percent": 3,
  "regression_limit_percent": 5,
  "repetitions": 10,
  "outliers": 1,
  "decision": "fail",
  "diagnostic_details": "[REDACTED]"
}

Do not log full environment dumps, source inputs, user data, arbitrary command arguments, or unrelated filesystem contents. Store detailed benchmark artifacts under normal repository access controls and keep the review record limited to fields needed to reproduce the decision.

Failure modes

Different baselines and testbeds. A candidate on one runner compared with a baseline from another can measure hardware or system state instead of code. Require matching testbed identity and build configuration.

One sample decides the gate. A transient scheduler delay can look like a regression. Require the declared number of repetitions and reject incomplete runs.

Statistical significance becomes the budget. A tool may detect a tiny repeatable change that has no practical impact, or fail to establish confidence in a large but noisy change. Evaluate both uncertainty and the project’s explicit regression limit.

The threshold is tuned after seeing the patch. Raising the limit to admit one result removes the gate’s independence. Change benchmark policy separately and regenerate an approved baseline.

Setup work leaks into timing. Fixture construction or unrelated I/O can dominate the measured function. Separate setup unless end-to-end latency is the intended contract.

The benchmark returns the wrong result quickly. Performance evidence without a correctness assertion rewards broken behavior. Make correctness a prerequisite and include its status in the evidence packet.

Outliers are silently discarded. Criterion.rs reports outliers because a large count is evidence of noise. Preserve the observations, investigate inconsistent work per iteration, and increase measurement time only through a declared policy change.

The baseline cannot be audited. A copied percentage without its source result, revision, tool version, and metric cannot support review. Keep machine-readable baseline and candidate artifacts together.

The agent edits the measuring stick. A patch can appear faster after shrinking the input or moving work outside the measured region. Review benchmark changes separately and compare the contract with the target branch before accepting results.

FAQ

How many repetitions should the gate use?

There is no universal count. Ten repetitions in the example policy are illustrative. Choose a minimum that produces stable results for the benchmark on the canonical runner, then store it in policy. Very fast or highly variable work may need longer measurement time, more repetitions, or a redesigned benchmark.

Should the gate compare means or medians?

Either can be useful, and the referenced tools report several statistics. The contract must name one decision statistic and use it for both revisions. Retain the other statistics and individual observations so reviewers can see variance and outliers.

Why rerun a failed result at all?

A single prescribed rerun distinguishes a repeatable code effect from a transient invalid run. The retry rule must be fixed in advance. Repeating until the test passes biases the evidence and should itself fail review.

Can a developer laptop provide the blocking result?

Local benchmarks are useful for exploration. A blocking decision should come from the controlled testbed that produced the compatible baseline. If no controlled runner exists, label the local result provisional instead of presenting it as a merge gate.

What should the coding agent receive after a failure?

Provide the benchmark policy, both complete result files, the correctness result, sanitized run context, the exact failure decision, and the relevant diff. Then write a focused task brief with reviewable boundaries . Do not ask the agent to make the number green by any means available.

Should benchmark changes and implementation changes share one patch?

Sometimes a new behavior requires both, but reviewers should still inspect them as separate logical changes. Establish the benchmark’s correctness and sensitivity first when possible. Any input, threshold, statistic, or timed-region change needs explicit justification.

Reader next step

Select one performance-critical behavior in your repository and write its benchmark contract today. Capture a target-branch baseline and a candidate result on the same controlled runner. Prove the happy path with an unchanged implementation, then introduce a temporary known slowdown to confirm that the error path blocks it. Revert that slowdown, retain the test artifacts, and require the same evidence shape for the next performance-sensitive coding-agent patch.