Playwright Flake Triage

npm CI

Playwright tells you a test is flaky. It doesn't tell you why, which flakes share a cause, or which one to fix first. This does.

Point it at your Playwright JSON reports and it groups every failure by root cause, ranks the causes by how many tests each one breaks, and tells you how to fix them.

Zero dependencies. No build step. Works offline.

npx playwright-flake-triage

🟠 Suite health 75/100 (unreliable)
   2/8 tests unstable · 8 failed attempts · 5 distinct causes · 4m 14s wasted

   Fix these first
   1. Cross-client / realtime event never arrived ← fixes 2 tests
      [user-b] No invite received within 45s
      3× · 2 test(s) · 2m 21s · 73f21adb
   2. Element not stable / not actionable
      locator.click: Timeout 30000ms exceeded.
      2× · 1 test(s) · 1m 2s · afc1e93a

---

Why this exists

Anyone who has run a large Playwright suite knows the two failure modes that waste the most time:

One cause, twenty red tests. A single race condition breaks tests across six files. The CI summary shows twenty unrelated-looking failures, so it gets triaged twenty times. Grouping by normalised failure signature collapses that back into one item — and tells you that fixing it fixes twenty tests.

The flake that never looks flaky. With retries: 0, a test that fails one run in fifteen is indistinguishable from a real regression. Every occurrence gets investigated from scratch. Playwright can't see this, because it only ever sees one run. This tool keeps history, so it can.

Everything here is aimed at one output: an ordered list of what to fix, with the reasoning shown so you can disagree with it.

---

Install

npm i -D playwright-flake-triage

Or don't install it at all — npx playwright-flake-triage works standalone.

You need the JSON reporter switched on:

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['list'],
    ['json', { outputFile: 'test-results/results.json' }],
  ],
});

Use

# Triage the latest run and remember it for cross-run detection
flake-triage

# Analyse a folder of archived CI reports, write a Markdown report
flake-triage ci-reports/ --no-history --out flake-report.md

# Gate a pipeline: fail the build if the suite is under 90% stable
flake-triage --min-health 90

Run it after every CI run and let it accumulate history. Cross-run detection needs at least two runs to say anything, and gets sharper with twenty.

Options

Flag	What it does
--out <file>	Write the Markdown report
--json <file>	Write the full analysis as JSON
--history <dir>	History location (default .flake-triage)
--no-history	Analyse only the given reports
--no-persist	Read history, don't record this run
--top <n>	How many causes to detail (default 10)
--min-occurrences <n>	Ignore one-off signatures
--min-health <n>	Exit 1 below this health score
--max-unstable <n>	Exit 1 above this many unstable tests

Exit codes: 0 fine, 1 a gate failed, 2 no usable report.

---

What it detects

Each root cause carries an explanation and a fix that resolves the problem rather than hiding it.

Cause	Usually
Strict-mode violation (ambiguous locator)	test debt
Element not stable / not actionable	test debt
Cross-client / realtime event never arrived	either
Browser or context closed mid-test	test debt
Fixed sleep in the failing path	test debt
Navigation or network failure	either
Locator never resolved	either
Assertion mismatch	either
Whole-test timeout	test debt

"Either" means the tool won't guess for you. A locator timeout can be a slow CI box or a genuine regression, and the honest answer is that the error text alone doesn't say. Where it can gather evidence — does this test pass in other runs, did it fail on several workers, does the actual value change between occurrences — it reports that instead of asserting a conclusion.

One distinction worth calling out, because nothing else seems to make it: a TimeoutError raised by Playwright's own retry loop and a bare Error: no message received within 30s thrown by your helper code are different problems. The first means an element never satisfied a condition. The second means somebody hand-rolled a wait for an event coming from outside the page, and in multi-client suites that's the single most common source of flake. They're classified separately.

---

How ranking works

priority = tests_affected × 10  +  occurrences × 2  +  wasted_minutes

Blast radius dominates deliberately. A cause breaking eight tests is worth more of your morning than one breaking a single test forty times, even though the second is louder.

Every term is printed next to the score, so you can argue with the ranking instead of trusting it. Confidence is reported per cause; below about 70%, open a trace before acting.

Signature grouping

Two failures from the same cause are almost never byte-identical — timeouts differ by milliseconds, ports rotate, temp paths carry a run id. Grouping on raw text scatters one cause across dozens of "unique" failures.

Before grouping, the message is stripped of timestamps, UUIDs, absolute paths, ports, query strings, hashes and bare numbers, and reduced to its first few lines. Selectors and assertion names survive, because those identify the cause. The result is hashed (FNV-1a) into a short stable fingerprint you can grep for across runs and reports.

---

Programmatic use

import { triage } from 'playwright-flake-triage';

const { result, markdown } = await triage(['test-results/results.json'], {
  historyDir: '.flake-triage',
  persist: true,
});

if (result.summary.suiteHealth < 90) {
  console.error(`Suite health ${result.summary.suiteHealth}`);
  process.exitCode = 1;
}

Lower-level pieces — analyze, classify, signatureOf, loadReports — are exported too, if you want to build a custom reporter or feed the analysis somewhere else.

CI

- run: npx playwright test
  continue-on-error: true

- run: npx flake-triage --out flake-report.md --min-health 85

- uses: actions/upload-artifact@v4
  with:
    name: flake-report
    path: flake-report.md

Cache or commit .flake-triage/ so history survives between runs — that's what powers cross-run detection.

---

Development

npm test     # 46 tests, node:test, no dependencies
npm run demo # run against the bundled fixtures

There's a worked example in examples/sample-report.md.

Licence

MIT