Last reviewed: 2026-09-09.

Direct answer

Expose test-order dependencies by running the affected test scope in a randomized order, preserving the seed and execution context, and replaying the exact order whenever it fails. A useful review gate has two modes: discovery chooses a fresh order; replay accepts the captured seed. The patch is ready only after the failing order passes with the fix and additional fresh orders also pass.

Treat the seed as part of the failure identity. Record it beside the commit, exact test selection, framework and version, operating system and architecture, worker count, exit code, and first failing test. Without that context, a reviewer may reproduce a different order and incorrectly conclude that the failure disappeared.

This check is especially valuable when a coding-agent patch changes fixtures, setup or teardown hooks, mutable process state, test databases, caches, temporary files, or shared mocks. Keep its scope in the patch plan alongside the broader change-scope note , so reviewers know which suites were randomized and which were not.

Who this is for

This workflow is for engineers who review agent-produced changes and operate their CI checks. It applies to maintainers of Go, Ruby, and JVM repositories, as well as mixed-language monorepos where each test runner exposes order controls differently.

Use it when tests pass in their normal sequence but fail intermittently as a suite, when a patch moves fixture code, or when a test succeeds alone and fails after another test. It is not a substitute for concurrency testing. If changing worker count or timing is the trigger, hold the order seed constant and follow a dedicated race-condition testing workflow .

Some integration and functional suites intentionally encode a sequence. JUnit’s documentation recognizes that explicit ordering can be necessary for those cases. Mark that contract clearly and keep it separate from tests that are expected to be independent. Randomizing an intentionally sequential workflow without acknowledging its contract produces noise rather than useful evidence.

Key takeaways

  • Run the unchanged baseline before judging the patch, then run the same selection in randomized order.
  • Capture the generated seed in machine-readable evidence and print a copy-paste replay command in the job summary.
  • During diagnosis, keep the commit, test selection, framework version, environment, and worker count fixed. Change one variable at a time.
  • Fix leaked state or incomplete cleanup instead of permanently pinning the accidental passing order.
  • Prove the repair against both the original failing seed and fresh seeds. One green random order covers only that sampled order.
  • Log an allowlist of diagnostic fields; do not archive whole environments or raw fixture payloads.

Sources checked

The Go command documentation defines -shuffle=off, -shuffle=on, and -shuffle=N. With on, the system clock seeds the randomizer; with an integer, that integer is the seed. In both cases, Go reports the seed for reproducibility.

The RSpec command-line order documentation explains that --order rand randomizes files, groups, and examples within the runner’s nesting rules. It also documents seeded forms such as --order rand:123 and the equivalent --seed 123 replay.

The JUnit test execution order guide documents deterministic default ordering, pseudo-random MethodOrderer.Random and ClassOrderer.Random implementations, and support for a custom seed. It also distinguishes method ordering from class ordering, which matters when shared state crosses class boundaries.

Together, these sources support a common contract: randomize at the framework’s supported boundary, retain the seed, and make replay a first-class CI operation. They do not establish a universal number of random runs, so teams should choose a budget based on suite cost and change risk.

Contract details to verify

Before adding the gate, define what must remain stable between discovery and replay:

  • Revision: run the same commit and dependency state.
  • Selection: preserve the exact packages, files, tags, filters, and exclusions.
  • Order scope: state whether the runner randomizes packages, files, classes, groups, methods, examples, or more than one level.
  • Seed: capture the value as data, not only as unstructured console text.
  • Execution shape: record worker count and whether parallel execution is enabled.
  • Environment: record a minimal runner fingerprint, including framework version, operating system, and architecture.
  • Result: retain the exit code, first failure, duration, and path to the complete test log.

Happy path

First, run the normal test command on the reviewed commit. Then run the same test selection with random ordering. In a shell job, preserve pipeline failures so that tee cannot turn a failed test process into a green step:

set -o pipefail
go test -shuffle=on ./... 2>&1 | tee test-order-go.log
bundle exec rspec --order rand 2>&1 | tee test-order-rspec.log

Use only the command for the repository’s framework. For JUnit, configure the built-in random orderers at the levels the suite needs:

junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$Random
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$Random

Supply a custom seed through the supported configuration of the JUnit launcher in use, and make the job print that value beside the test result. Do not assume method randomization also changes class order; the guide exposes separate method and class orderers.

On success, store an allowlisted record similar to this one:

{
  "event": "test_order_check",
  "commit": "8f3a21c",
  "framework": "go",
  "framework_version": "go1.27.1",
  "test_selection": "./...",
  "order_mode": "random",
  "seed": 123,
  "attempt": 2,
  "worker_count": 1,
  "runner_os": "linux",
  "runner_arch": "amd64",
  "exit_code": 0,
  "result": "passed",
  "first_failure": null,
  "log_path": "artifacts/test-order-go.log"
}

Keep environment values, user data, and fixture contents out of this record. If a diagnostic field can contain sensitive text, store a fixed redaction marker or omit the field. After the first randomized pass, spend the job’s remaining risk-based budget on fresh seeds. Save the generated replay command even when the run passes; consistent evidence makes later failures easier to compare.

Error path

When discovery fails, do not let an automatic retry replace the original result. Preserve the log and seed, then replay the same order on the same commit:

go test -shuffle=123 ./...
bundle exec rspec --seed 123

Again, select only the command for the repository. For JUnit, reuse the recorded custom seed through the same launcher configuration used during discovery.

If the replay fails consistently, reduce the case without changing several controls at once. Run the failing test alone. Then run it with likely predecessors in both orders. Inspect setup and teardown around the smallest pair that changes the result. Common suspects include global registries, process-wide configuration, reused database rows, shared mock expectations, cached singletons, fixed temporary paths, and random generators reset by another test.

Repair ownership of that state: initialize it in the test that needs it, restore mutations during teardown, allocate isolated resources, and assert cleanup where practical. Rerun the original seed first. Then rerun the default order and several fresh random orders. Attach all three outcomes to the review rather than showing only the final green attempt.

If the same seed does not reproduce the failure, keep the seed fixed and investigate one additional dimension at a time: worker count, operating system, framework version, time, filesystem state, or an external dependency. This is where a disciplined CI repair loop prevents unrelated edits from obscuring the original signal.

Failure modes

  • Randomizing without retaining the seed. The failure becomes a screenshot rather than a repeatable test case.
  • Replaying against a different revision. A matching seed does not compensate for changed test discovery, dependencies, or source code.
  • Changing the selection during replay. Filters can alter the available tests and therefore the meaning of an order, even when the numeric seed is unchanged.
  • Changing order and parallelism together. Two moving variables make a failure harder to classify. Hold the worker count constant until order dependence is confirmed or rejected.
  • Randomizing only one level. Method order can be clean while shared state leaks between classes. Verify the actual scope provided by the selected runner configuration.
  • Treating a retry as the result. A green second attempt must not erase the first failure, its seed, or its log.
  • Pinning the accidental passing order. Explicit order is valid when sequence is the stated test contract. It is a weak repair when independent tests merely happen to pass in one sequence.
  • Dumping excessive diagnostics. Whole environment snapshots and raw fixtures expand exposure without making the order reproducible. Prefer the small field allowlist shown above.
  • Declaring victory after one fresh seed. The original failure proves at least one harmful order exists; a single different order says little about whether cleanup is complete.

FAQ

Is one passing randomized run enough?

No. It is useful evidence for one sampled order, not proof over every possible order. Always replay a known failing seed after the repair, then add fresh seeds according to the suite’s cost and the patch’s risk.

Should every test suite be randomized?

Randomize suites whose contract requires order independence. Keep intentionally sequential integration flows explicit and documented. In a large repository, start with the test scope touched by the patch, then expand once runtime and failure evidence are manageable.

Why can the same seed produce a different result?

The seed controls only the ordering mechanism implemented by the runner. A changed test selection, framework version, worker count, revision, or environment can still change the run. Compare the full recorded contract before concluding that seeded replay is broken.

Should a reviewer accept an order annotation as the fix?

Only when the sequence is intentional behavior of the test scenario. For tests that claim independence, an order annotation can hide leaked state. Ask for the state owner, isolation boundary, and teardown behavior to be made explicit.

Does this replace other test-strength checks?

No. Order randomization targets contamination between tests. Property-based testing explores generated inputs, mutation testing evaluates whether assertions catch injected faults, and race testing targets concurrency. Apply each technique to the failure class it can actually expose.

Reader next step

Choose one test target changed by the patch. Add a normal-order run and one randomized discovery run, capture the seed and the allowlisted fields above, and make the job print an exact replay command. Trigger the error path with a genuine observed failure rather than an invented result. Before merge, require evidence that the fixed revision passes the original seed, the default order, and the agreed fresh-seed budget.

Then include the commands, seed, revision, and log paths in the pull-request review handoff . That gives the next reviewer a reproducible check instead of a claim that an intermittent failure could not be found again.