Last reviewed: 2026-08-10

Direct answer

Score a coding-agent patch as a small set of independently inspectable dimensions, then combine those scores with an explicit policy. A useful starting rubric is behavior (0.50), test evidence (0.30), and reviewability (0.20), with every dimension graded from 0 to 1. The weights are an operator policy, not a claim about any vendor’s defaults. Keep the component scores, evidence, and reason codes instead of storing only one pass or fail bit.

This approach matters because a patch can be partly right. It may implement the requested behavior but omit a regression test; it may pass the visible tests while changing an unrelated interface; or it may be blocked by a broken environment before the agent gets a fair chance. A partial score makes those distinctions visible. It does not lower the merge bar for a critical defect. Treat a security, data-loss, or compatibility violation as a hard stop even when the arithmetic score is high.

OpenAI’s Graders guide describes graders that compare a reference answer with a model-generated answer and return a value from 0 to 1. It explicitly notes that partial credit can be more useful than a binary result. For a coding agent, the reference can be an expected diff, an acceptance-test outcome, or a structured set of invariants. The grader should answer one narrow question at a time, such as “does the patch preserve the public function signature?” rather than trying to judge the whole change in one opaque prompt.

Who this is for

This workflow is for maintainers, platform engineers, and reviewers who run coding agents against issues, pull requests, or a queue of repeatable maintenance tasks. It fits a local harness, a CI job, or an ephemeral cloud workspace. It is especially useful when more than one model, prompt revision, tool set, or sandbox image is being compared and anecdotes are no longer enough.

It is not a replacement for a human code owner. A score is a compact decision aid. The owner still decides whether the change fits product intent, security policy, and the repository’s unwritten constraints. Small teams can start with five or ten representative fixtures; larger teams can stratify fixtures by language, framework, change size, and risk.

Key takeaways

  • Define a fixture before asking an agent to edit: issue text, base revision, setup command, acceptance checks, and an explicit out-of-scope list.
  • Grade separate dimensions. Keep behavior, tests, reviewability, and policy gates distinct so one strong signal cannot hide a critical weakness.
  • Run the same fixture in a controlled environment and record the revision of the repository, prompt, model, grader, and environment.
  • Use thresholds as routing rules: merge candidate, human review, retry, or blocked. Do not silently turn a missing signal into a zero.
  • Compare distributions and failure reasons across repeated runs, not just the average score of one run.

The OpenAI evaluation best-practices guide warns that generative outputs vary and recommends purpose-built evaluations, automated scoring where possible, and human judgment alongside numerical metrics. That combination is the central design principle here: automation narrows the review surface, while a person retains authority over ambiguous or high-impact decisions.

Sources checked

The OpenAI Graders documentation supports the 0-to-1 grading model, reference-versus-output comparisons, JSON-defined grader types, and partial credit. The OpenAI evaluation best-practices documentation supports the need for purpose-built tests, automated scoring, continuous evaluation, and human judgment.

The SWE-bench repository is a practical public example: it evaluates patches for real GitHub issues and uses Docker for reproducible evaluations. Its documented harness produces build logs, evaluation logs, and result files, which is a useful model for separating raw evidence from the final score.

The GitHub Copilot cloud agent documentation describes an ephemeral development environment where an agent can research a repository, plan, edit a branch, run tests and linters, and leave commits and logs that people can review. That makes branch and log provenance part of the evaluation record, not an afterthought.

Finally, Anthropic’s Building Effective AI Agents recommends simple composable workflows, ground truth from the environment, checkpoints, stopping conditions, and sandboxed testing with guardrails. It also describes an evaluator-optimizer loop as a good fit when criteria are clear and iteration produces measurable improvement. These sources are independent: they describe different products or research projects, but converge on controlled tasks, explicit criteria, and inspectable evidence.

Contract details to verify

Before implementing a grader, write down the contract it is supposed to enforce. A fixture should include:

  1. Task identity. Use a short stable task_id, the repository name, and the exact base revision. Do not use a moving branch as the baseline.
  2. Expected behavior. State inputs, outputs, error behavior, and compatibility constraints. If there is a reference patch, keep it separate from the agent’s workspace.
  3. Evidence sources. List the tests, static checks, diff inspection rules, and human questions that contribute to each dimension. A test result is evidence, not the entire definition of correctness.
  4. Scoring math. Document weights, missing-signal handling, rounding, and hard gates. For example, a behavior score of 0.8, test score of 0.6, and reviewability score of 1.0 yields 0.50*0.8 + 0.30*0.6 + 0.20*1.0 = 0.78.
  5. Decision bands. Pick bands that match your risk tolerance. One illustrative policy is 0.85 or higher for a merge candidate, 0.60 through 0.84 for human review, and below 0.60 for retry or rejection. A hard gate can override every band.

If you use a hosted grader, verify the current API, SDK, retention, and deprecation terms before building a release dependency. The current OpenAI documentation says that graders are being deprecated as part of the evals and fine-tuning workflows, so treat the guide as evidence for the grading concept while confirming the supported production path. Keep a local adapter around your rubric so a service transition changes an integration rather than your policy.

A good contract also says what the grader must not see. Give it the fixture, patch, and allowed test outputs; exclude unrelated repository history, hidden answer keys, and credentials. If an evaluator model is used, keep its instructions independent from the agent’s task prompt and sample enough runs to expose variance.

Failure modes

Binary scoring hides useful progress. A patch that fixes the main branch but misses one edge case can look identical to a patch that does nothing. Preserve component scores and a short reason code such as missing-regression-test or scope-expansion.

A passing test suite is over-trusted. Tests can be incomplete, incorrectly configured, or too close to the visible example. Add behavior invariants, diff checks, and a human question for high-risk changes. Property-based or mutation testing can be a useful companion; the allowed property-based testing guide covers a separate way to probe edge cases.

Environment drift changes the result. Different dependency locks, operating systems, or service versions can turn the same patch into different outcomes. Pin the base revision and sandbox image, record setup failures, and make the environment a first-class fixture input. SWE-bench’s Docker harness is a concrete example of this reproducibility goal.

Nondeterminism is mistaken for regression. An agent may choose different files or tools on two runs. Run a small repeated sample, report the range and failure reasons, and compare like-for-like configurations. Do not declare a model winner from a single lucky trajectory.

The grader rewards the wrong behavior. A patch can satisfy a narrow string check while breaking an API, deleting a needed test, or expanding scope. Use several narrow graders, add hard policy gates, and periodically have reviewers inspect disagreements between the aggregate score and the final decision.

Missing evidence becomes a misleading zero. A timeout, unavailable test service, or truncated tool trace is an execution status, not proof that the patch failed. Emit blocked or inconclusive, preserve the reason, and route it to an operator. Retrying the same broken environment only creates noise.

Sensitive data leaks through logs. Store identifiers, revisions, counts, booleans, scores, and reason codes. Do not store prompts, file contents, environment variables, or authentication material in the evaluation record. Sanitization should happen before logs leave the sandbox.

For a concrete happy path, create the fixture, launch the agent in the pinned sandbox, run the acceptance checks, calculate each component score, and route the result according to the documented bands. For the error path, stop when setup or a hard gate fails, record the sanitized status, preserve the partial evidence, and ask a human whether to repair the fixture or retry the task. This keeps operational recovery separate from patch quality.

Here is a compact record shape that contains useful evidence without secrets or patch text:

run_id: run-042
task_id: issue-17
repository_revision: abc1234
environment_revision: image-7
model_id: candidate-a
prompt_revision: prompt-3
grader_revision: grader-2
tool_calls: 7
tests_passed: 18
tests_failed: 2
behavior_score: 0.75
test_score: 0.60
reviewability_score: 1.00
decision: manual-review
reason_code: missing-regression-test

FAQ

Should every patch receive a partial score? Only when the relevant evidence exists. If the sandbox never started or the trace is incomplete, use blocked or inconclusive and explain why. A fabricated zero contaminates comparisons.

What should the reference answer be? Use the smallest stable representation that captures the contract: expected outputs, invariants, acceptance-test results, or a reviewed reference diff. Avoid requiring the agent to reproduce formatting that has no product value.

Can an evaluator model grade code? It can help with criteria that are difficult to encode mechanically, but keep the rubric narrow, compare its judgments with human samples, and retain deterministic checks for compilation, tests, and policy. Anthropic’s evaluator-optimizer pattern is most appropriate when criteria are clear and improvement is measurable.

How many runs are enough? There is no universal number. Start with repeated runs on a representative fixture set, then increase the sample when scores are close, variance is high, or the change is risky. Track the same fixture and environment revisions so the comparison remains meaningful.

Do these scores decide a pull request automatically? They can route routine work, but a hard gate or an ambiguous result should require a code owner. GitHub’s cloud-agent workflow illustrates why commits, logs, tests, and branch state should remain reviewable before a pull request is accepted.

Reader next step

Pick one recurring issue type and create three fixtures from recently reviewed patches. For each fixture, record the base revision, acceptance checks, one behavior invariant, and one reviewability question. Run the current agent twice in the same sandbox, calculate the three component scores, and compare the reason codes before changing the prompt or model.

Then document the evidence trail using Agent Run Evidence Ledgers for Human Review and stress the acceptance checks with Find Edge Cases in Coding Agent Patches With Property-Based Testing . The goal of this first pass is not a perfect leaderboard. It is a repeatable decision record that tells the next reviewer what passed, what only partly passed, what could not be measured, and what should happen next.