Last reviewed: 2026-08-06

Direct answer

Property-based testing gives a coding-agent patch a behavioral contract instead of a short list of hand-picked examples. You describe an invariant that should hold for every valid input, let a test library generate many inputs, and preserve the smallest input that violates the invariant. That smallest failure is much more useful to an agent than a vague instruction such as fix the edge cases.

The Hypothesis documentation describes this Python workflow directly: tests state the range of acceptable inputs, and Hypothesis chooses inputs that include edge cases a developer may not have anticipated. For JavaScript and TypeScript, fast-check generates cases and shrinks a failing case to the smallest input that still reproduces it. The important operating pattern is the same in either language: write the property first, keep the property independent from the implementation, and send the minimized failure back as a bounded repair task.

Suppose an agent changes a function that sorts integer lists. The examples may cover an already sorted list and a list with a duplicate. A property can cover the rules that matter for every list: the result is ordered, the result has the same length, and the result contains the same values with the same multiplicities.

from collections import Counter

from hypothesis import given, strategies as st

from app.sorting import sort_numbers


@given(st.lists(st.integers()))
def test_sort_numbers_properties(values):
    result = sort_numbers(values)
    assert len(result) == len(values)
    assert Counter(result) == Counter(values)
    assert all(left <= right for left, right in zip(result, result[1:]))

This test is an oracle, not a second implementation of the sort. It does not calculate the expected sorted list and then compare two implementations that could share the same mistake. It checks independent consequences of the contract. If the agent accidentally drops a duplicate, mishandles an empty list, or returns an unsorted boundary pair, the property can expose the defect.

The research paper Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback describes the same feedback shape for automated code refinement. Its Property-Generated Solver checks high-level properties and gives the model a structurally minimal failing counterexample. The authors report benchmark improvements, including up to a 13.4 percent pass-at-one improvement and a fix rate above 64 percent on initially failed problems. Treat those as findings from that study, not as a promise for every repository. The practical lesson is narrower and useful: a precise property plus a small reproducer gives an agent a cleaner repair signal than a noisy test transcript.

A happy-path and error-path operator workflow

  1. Write the contract before asking for the patch. Record the valid input domain, preconditions, invariants, unacceptable side effects, and any performance boundary. A focused coding-agent task brief can keep those constraints visible to the agent.
  2. Build the property and establish a baseline. Run it against the approved revision before the agent changes production code. A failure at baseline means the property or harness needs attention; it is not evidence against the new patch.
  3. Give the agent a narrow repair task. Allow implementation changes, but keep the property and its generator human-owned unless a separate review explicitly changes the contract. Tell the agent to reproduce the reported case, change the implementation, and leave the property intact.
  4. On the happy path, run examples and generated checks. A passing run means the patch survived the configured examples and property cases. Review the diff, retain the property as a regression test, and record the framework version and configured case budget.
  5. On the error path, preserve the minimized failure. Capture the property identifier, test path, framework version, replay seed when available, exception class, and minimized input or a safe shape of it. Ask the agent to reproduce that exact failure before it proposes a fix. After the repair, rerun the saved case and then run fresh generated cases.
  6. Stop the loop when the evidence stops improving. If the failure cannot be reproduced, the property is flaky, or the agent keeps weakening the oracle, pause for human review instead of increasing retries indefinitely. A reviewable follow-up from pull request feedback is a better handoff than an unbounded repair loop.

Who this is for

This workflow is for engineers who use coding agents to change deterministic business logic: parsers, normalizers, serializers, validators, scheduling rules, collection transformations, and other functions with a describable input domain. It is especially useful when a patch looks simple but has many combinations of values that examples cannot cover.

It is also for reviewers who need evidence stronger than a green example-based test run. The reviewer does not need to accept every generated case as a release gate. The useful discipline is to make the invariant, generator limits, replay information, and ownership of the test explicit.

Start with one function and one property. Do not begin by asking an agent to invent a broad testing strategy for an entire repository. For JavaScript and TypeScript, fast-check provides a direct implementation path. For Python, Hypothesis provides the same core pattern. The current jqwik guide documents a Java implementation with generation, edge cases, and shrinking, but it also includes an explicit policy saying the library is not intended for AI coding agents. That restriction must be honored; do not direct an agent to use jqwik or hide that policy from a reviewer.

Key takeaways

  • A property states an invariant; it is not a pile of expected outputs.
  • Baseline the property before the patch so a pre-existing failure is not misattributed.
  • Treat the minimized counterexample as the primary repair input.
  • Keep generators inside the real domain and bound their cost.
  • Let the agent repair implementation code without silently editing the oracle.
  • Store replay metadata and safe failure shapes, not production payloads.
  • Check each testing library’s current usage terms before placing it in an agent workflow.

Sources checked

  • arXiv paper on property-oriented LLM refinement : the May 2026 revision describes semantic properties, structurally minimal counterexamples, and measured code-refinement results.
  • Hypothesis 6.165.2 documentation : the current Python documentation explains generated inputs, unexpected edge cases, and property assertions such as ordering and membership.
  • fast-check introduction : the current JavaScript and TypeScript documentation explains generated cases and shrinking to a smallest reproducer; the page was updated in April 2026.
  • jqwik User Guide 1.10.1 : the Java guide documents parameter generation, edge-case generation, result shrinking, replay information, and a current explicit anti-AI usage policy.

These sources support the testing mechanics and the shape of an agent feedback loop. They do not establish that any particular generated test proves production correctness. That decision remains a repository-specific engineering judgment.

Contract details to verify

Before the agent starts, verify the following details in the task record:

  • Input domain: Which values are valid, and which are rejected before the property runs? Generating invalid values can turn a useful property into a misleading failure.
  • Invariant and oracle: Can a reviewer explain why the assertion is independent of the changed implementation? Prefer conservation, ordering, round-trip, monotonicity, or idempotence rules that do not copy production logic.
  • Generator boundaries: What sizes, encodings, nullability rules, and special values are in scope? Set a case budget that fits the test job and make rejected values visible in the report.
  • Replay contract: Which seed, framework version, configuration, and test path are required to replay a failure? A seed without the matching environment is incomplete evidence.
  • Change ownership: Is the agent allowed to edit only implementation files, or may it propose a separately reviewed property change? Make this explicit before a failure occurs.
  • Data handling: Can the minimized input be committed? If it could contain customer data, credentials, or private repository content, record only a sanitized shape and recreate a synthetic fixture.
  • Tool policy: Does the framework permit the intended automation? The jqwik source demonstrates why this check belongs in the contract rather than being discovered after an agent has started.

A sanitized event record can preserve replay and routing information without copying the input itself:

run_id: pbt-042
patch_revision: local-3
property_id: sort-order-and-membership
framework: hypothesis
framework_version: 6.165.2
case_count: 200
outcome: failed
seed: 1842
shrunk: true
counterexample_shape: integer-list-length-2
failure_type: invariant-mismatch
test_path: tests/test_sort_properties.py
duration_ms: 384
data_classification: synthetic
next_action: repair-implementation

The values above are illustrative. A real record should exclude request headers, environment values, customer payloads, and any credential material. If the minimized case is safe and synthetic, commit it as a named regression fixture. If it is not safe, retain only the shape and a reproducible generator configuration in the review packet.

Failure modes

  • The property mirrors the patch. If the agent can change both sides of the comparison to the same incorrect rule, a green run has little value. Keep the oracle conceptually separate.
  • The generator leaves the domain. Invalid or unrepresentative inputs produce failures the product never promised to handle. Encode preconditions and explain rejected values.
  • The property is too weak. A test that checks only that a function returns without raising can pass while data is lost or reordered. Add the business invariant that matters.
  • The agent edits the test to remove the failure. A passing run after an oracle change is not equivalent to a passing run after an implementation repair. Review test diffs separately.
  • The smallest case is discarded. A full failing transcript creates noise and makes reproduction harder. Preserve the shrunk case, seed, and exact property identifier.
  • Randomness becomes flakiness. Time, locale, concurrency, external services, and unstable ordering can make a property fail inconsistently. Isolate those dependencies or stop and escalate.
  • Logs expose the input. Generated values can contain private records even when the test author did not expect them. Scrub before central logging and use synthetic fixtures where possible.
  • The repair loop has no stop condition. Repeatedly asking for another fix can produce weaker properties and larger diffs. Set a retry cap and escalate when the invariant or environment is uncertain.
  • A library policy is ignored. A technically capable framework is not automatically approved for an AI workflow. Respect explicit usage restrictions such as the one in the current jqwik guide.

FAQ

Do property-based tests replace example-based tests?

No. Examples communicate named business scenarios and make a regression easy to read. Properties explore combinations that the author did not enumerate. Keep both when each catches a different class of mistake, and use the property failure to add a small, readable example when that improves future review.

How many generated cases should a gate run?

There is no universal number. Start with a bounded budget that completes reliably in the normal test job, then increase it when the property is cheap and the input domain is broad. Record the configured budget so a reviewer can distinguish a small smoke run from a full verification run. Do not treat a framework default as a guarantee of coverage.

Can the agent write the property?

It can draft a property from a written contract, but a human should verify the invariant, generator, and oracle before using the result as a gate. The agent should not be allowed to make a failing property pass by weakening its assertions without an explicit, separately reviewed contract change.

What should I send the agent after a failure?

Send the property identifier, exact test path, framework and version, replay seed if available, exception type, and the minimized input when it is safe. Include the expected invariant and a clear instruction to repair implementation code first. Do not paste an entire log when a small reproducer is available.

What if the failure cannot be reproduced?

Stop treating it as ordinary patch feedback. Compare the framework version, configuration, seed, locale, time source, and external dependencies. If the result remains unstable, isolate the nondeterministic dependency or escalate for human diagnosis rather than accepting a one-off green run.

How does this work across languages?

Use the property pattern rather than copying syntax. Hypothesis documents the Python approach, fast-check documents the JavaScript and TypeScript approach, and jqwik documents Java generation and shrinking. For jqwik specifically, follow its current anti-AI usage policy and do not place the library under an AI coding agent’s control.

Reader next step

Choose one small deterministic function changed by a recent coding-agent patch. Write three invariants and one explicit input boundary before asking for another edit. Add a property test, run it against the approved revision, and then give the agent only the minimized failure and the repair constraints. Keep the property in the diff, preserve a safe replay record, and review the resulting change with the same discipline as any other patch.

For a repeatable handoff, use the reviewable-diff workflow after the property run and the focused pull-request follow-up . The immediate goal is not to make an agent generate more tests. It is to make one important behavioral rule executable, reproducible, and hard to weaken accidentally.