Your Tests Pass in an Order Nobody Chose: Make Independence a Checked Property

On October 7, Playwright 1.64 shipped a small command line option. According to the release notes, --shuffle “schedules tests in a random order,” prints the seed it used, and accepts that seed back so the same order can be replayed. It is easy to read past a line like that. It is worth stopping on, because it hands every team running a browser suite a cheap way to answer a question many suites are never asked: do these tests pass because the code works, or because they happen to run in the order they always have?

Test tooling usually treats each test as an independent experiment. Reports, retries, sharding, parallel workers and test selection all lean on that working assumption, and it is worth checking rather than trusting. The order a suite runs in is usually an accident of file names, declaration order and runner internals, and code quietly comes to depend on accidents. This article looks at what the research says about order-dependent tests, why the order changes under you even when nobody touches the tests, what the major runners now offer, and a staged way to turn independence from an assumption into a property your pipeline checks.

What order dependence looks like in real suites

A 2014 empirical study, An Empirical Analysis of Flaky Tests by Luo, Hariri, Eloussi and Marinov at FSE 2014, examined 201 commits that likely fixed flaky tests across 51 open-source projects. Test order dependency was one of the three most common root causes, alongside asynchronous waits and concurrency, accounting for 19 of the 161 commits the authors could classify, or 12 percent. Two of its findings matter more than the share. First, 47 percent of the order-dependent cases came from dependence on external resources such as files and databases, not only from in-memory state like static fields. Second, 74 percent were fixed by cleaning the shared state between test runs. The problem is common, it is not confined to sloppy use of globals, and its usual cure is mundane.

The paper describes the two shapes the dependency takes. In the first, a “polluter” test changes shared state, and a later test that expected the original state fails. In the second, a test silently relies on state that another test was supposed to create, and fails whenever that other test has not run first. Its example comes from Hadoop: one test wrote a file that several read tests needed, and the developers assumed it would always run first, although JUnit guarantees no particular order. They fixed it by moving the write into class setup.

A newer study shows the same pattern in the JavaScript ecosystem, where many Playwright users also write their unit tests. Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript by Hashemi, Tahir, Rasheed, Shi and Blagojevic, accepted at ICST 2025, reordered the Jest suites of 81 GitHub projects at three levels, files, describe blocks and individual tests, with 10 unique orders per level and each order rerun 10 times. It found 55 order-dependent tests in 10 projects. Forty-two came from shared mocking state, a cause the authors say had not been reported before, and 13 from shared files. The fix was again unglamorous: clearing mocks in a beforeEach hook repaired 34 of the 39 mock-related cases the authors addressed.

Both studies sampled open-source projects, so neither tells you how many order-dependent tests your own suite contains. What transfers is the mechanism. Any suite that shares a database, a file system, a mock registry, a module-level cache or a logged-in session across tests has the raw material for the problem.

Why the order changes even when the tests do not

The usual defense is that the suite has been green for years, so its order must be fine. That holds only as long as the order holds, and the order is rarely a contract.

Runners change it. The Luo paper notes that the move from Java 6 to Java 7 changed the order in which JUnit discovered tests within a class, because the reflection library changed. JUnit Jupiter documents its default method order as “deterministic but intentionally non-obvious”, which is a polite warning not to build on it.

Parallelism changes it. Playwright’s parallelism documentation states that test files run in parallel by default while tests within a single file run in order in the same worker process, and that there is no guarantee about execution order across files. Turning on fullyParallel, adding workers, or splitting a run into shards each produces a different set of neighbors for every test. A test that was always preceded by the same setter in one worker may now land in a worker where that setter never ran.

Growth changes it. Adding a test file, renaming one, or moving a test between files reshuffles what runs before what. Test impact analysis and other selection schemes change it most aggressively of all, because they run subsets, and a subset is a different order with some tests deleted. A suite that is green only in full, in its historical order, will produce failures that appear on unrelated pull requests and look like random noise. That is the expensive form of the problem: the person who sees the failure is often not the person who introduced the dependency, and the failure gets retried rather than understood.

What the runners now give you

Seeded shuffling is no longer a research tool. Most mainstream runners offer it, with one design choice in common: a printed seed that replays the same order. The scope differs, and the scope determines which dependencies a shuffle can expose.

JUnit reports its seed differently from the others. Its MethodOrderer.Random documentation says the default seed is logged at CONFIG level and can be fixed through a seed configuration property. CONFIG sits below the level many builds display, so check that your build output actually contains the seed before you rely on replay.

Two details in that table deserve attention. The JavaScript study found 52 of its 55 order-dependent tests by reordering individual tests, three by reordering describe blocks, and none by reordering whole files. If your runner only shuffles at the file level, a clean result tells you little about dependencies between tests in the same file, and that is where the JavaScript study found almost all of its cases. And pytest-randomly’s reset of the global random seed is a reminder that “order” covers more than which test runs first: a test that passes only because an earlier test consumed a particular number of random values is order-dependent too.

A rollout that does not punish the wrong people

Switching shuffling on for every pull request on day one is the obvious move and usually the wrong one. If the suite already contains order-dependent tests, a random subset of unrelated pull requests will fail, authors will retry until green, and the team will learn to distrust the very signal you added. A staged rollout keeps the cost on the people fixing the problem.

Measure first, off the merge path. Add a scheduled job that runs the suite against the main branch with several different seeds, recording every seed that produced a failure. Nobody’s pull request is blocked, and after a few weeks of runs you have an inventory rather than an anecdote. If your CI already reruns failed tests automatically, make sure this job does not, because a retry that passes in a different order erases exactly the evidence you are collecting.

Separate order from other flakiness. For each failing seed, replay the same seed several times, and run the default order alongside it. A failure that reproduces consistently under one seed and not under others points toward an order dependency. A failure that comes and goes under the same seed points toward the other two large categories in the Luo study, asynchronous waits and concurrency, or toward external state such as a shared database. Treat these as clues for where to look first, not verdicts: a test can depend on order and on timing at once, so keep both explanations open until a minimal reproducer settles it. Lumping every failure into one list makes it hard to fix any of them.

Find the pair, then fix the state. Run the failing test alone. If it also fails alone, check first whether it depends on setup that another test used to provide; if it passes alone, look for an earlier test that polluted its state. Either way, confirm the hypothesis by narrowing the order to the smallest prefix that still fails, often a single earlier test, rather than trusting the first guess. Most fixes live in setup and teardown: reset the shared state, clear mocks, create the data a test needs inside that test or its fixture, and stop relying on one test to prepare another. Both studies point the same way: state cleanup fixed 74 percent of the order-dependent cases in the older study and 34 of the 39 mock-related cases in the newer one.

Gate once the nightly run is quiet. When scheduled shuffled runs have stayed clean for an agreed period, enable shuffling on pull requests, with the seed printed in the failure output and the exact replay command in your contributing guide. A quiet nightly history makes it more likely that a new failure relates to the change under review, but it does not prove it, since an order the nightly runs never sampled can still surface. Keep triage with the team that owns the suite rather than defaulting the blame to the author, and give authors a documented path to report a failure they believe is pre-existing.

Keep the exceptions explicit and few. Some sequences are legitimately ordered, such as a browser journey that signs up, verifies and then purchases. Express those as a single test with steps, or use the runner’s explicit mechanism. Playwright offers a serial mode for inter-dependent tests, and its own documentation says that “using serial is not recommended.” An explicit, reviewed exception is honest. An implicit one that only holds in the historical order is a defect waiting for a runner upgrade.

Tradeoffs worth naming

Shuffling costs very little runtime on its own; running several seeds nightly multiplies that job’s compute, which is the real budget decision. On large browser suites, start with a small number of seeds and widen it only if new failures keep appearing.

Shared expensive fixtures are a common source of tension. Teams reuse a database, a seeded data set or an authenticated session across tests because creating them per test is slow. Shuffling does not forbid that; it forces the shared resource to be either read-only or reset to a known state, and it exposes the cases where neither is true. Sometimes the answer is to make the shared fixture immutable and give each test its own writable slice, for example a unique tenant or schema, rather than to recreate everything.

Worker reuse is a subtler interaction. Playwright reuses a worker process across tests where it can, and its documentation notes that workers are always shut down after a test failure to guarantee a pristine environment for the tests that follow. That design is good for reliability, but it also means that module-level state and worker-scoped fixtures persist across consecutive passing tests in the same worker, which is where pollution can accumulate. Shuffling changes which tests share a worker, which makes it a cheap way to surface that.

Finally, not every suite needs this. A small library whose unit tests share no state, or a suite that is about to be replaced, will not repay the effort. The return is highest in integration and end-to-end suites that touch databases, file systems, caches or mocks, and in any organization moving toward parallel execution, sharding or test selection, because each of those quietly changes the order anyway.

If your test suite fails on changes that did not cause the failure, or you are about to parallelize, shard or select tests, we help engineering teams inventory hidden test dependencies with seeded runs, separate order problems from timing and concurrency flakiness, and design fixtures and pipeline gates that keep the suite trustworthy as it grows. Talk with us about making your test pipeline reliable.