Your E2E tests pass. Are you sure they can fail?
Someone debugging a test in code-server, the 78k-star VS Code-in-the-browser project, did what all of us do: they focused a single test to iterate faster. it.only("should change to expired when not active", async () => { Then it got committed. For roughly seven months, that one .only silently disabled the other eight tests in the file. CI stayed green the entire time, because from CI's point of view nothing was wrong: the focused test ran, it passed, done. By the time the suite was re-enabled, o
Someone debugging a test in code-server, the 78k-star VS Code-in-the-browser project, did what all of us do: they focused a single test to iterate faster.
it.only("should change to expired when not active", async () => {
Enter fullscreen mode Exit fullscreen mode
Then it got committed. For roughly seven months, that one .only silently disabled the other eight tests in the file. CI stayed green the entire time, because from CI's point of view nothing was wrong: the focused test ran, it passed, done. By the time the suite was re-enabled, one of the skipped tests had rotted and no longer passed. For over half a year, the only signal anyone got was a green checkmark.
The fix is merged now (coder/code-server#7845, which also repaired a batch of one-shot isVisible() reads and matcher-less expect() calls in the E2E suite). To be clear, this is not a story about code-server being sloppy; it is one of the better-maintained projects I have reviewed. That is what makes it a useful opening exhibit. If a leaked .only can sit unnoticed for seven months in a repo like that, it can sit unnoticed anywhere.
The same bug in four more projects
Once I started looking for tests that pass without proving anything, I kept finding them in heavily maintained, widely used codebases. Here is the whole genre in one merged one-line fix, from Carbon Design System (carbon#22564):
- expect(page.locator('.cds--progress-step--complete')).toBeTruthy();
+ await expect(page.locator('.cds--progress-step--complete')).toBeVisible();
Enter fullscreen mode Exit fullscreen mode
page.locator() returns a Locator handle no matter what is on the page. The element can be missing or the whole feature deleted; the handle is still truthy. The original assertion has never failed and never will.
Ghost had the async variant (Ghost#28712):
expect(likeButton.isDisabled()).toBeTruthy();
Enter fullscreen mode Exit fullscreen mode
isDisabled() returns a Promise, and a Promise is always truthy. Whether the button is disabled, enabled, or not rendered at all, this passes.
Strapi had checks computed and then thrown away (strapi#26630): isVisible(), isHidden(), isEnabled() called, their results read into thin air, never asserted. SvelteKit had web-first assertions missing their await (kit#16068), so the assertion floated off and never participated in the test outcome.
Different-looking bugs, one failure mode: the test runs, goes green, and the green means nothing. I started calling these silent-pass tests. Their dangerous property is camouflage. A missing test at least looks like a gap; a silent-pass test looks like coverage. They are also not rare. In the 100-PR benchmark corpus described below, 33 of the 100 PRs touching Playwright or Cypress specs contained at least one genuine issue of this class.
Check your own suite in the next five minutes
You do not need my tool for the mechanical cases. You need grep. These four are triage: they will surface some legitimate lines (a .toBeDefined() on a config object is fine; on a Locator it is decorative), and each hit takes a few seconds to eyeball. They are written for Playwright; Cypress variants live in the repo.
# 1. Committed focused tests: CI runs one test, silently skips the siblings
grep -rnE "(it|test|describe)\.only\(" e2e/
# 2. Truthiness asserted on a Locator handle: always passes
grep -rnE "expect\([^)]*\)\.(toBeTruthy|toBeDefined|not\.toBeNull)\(" e2e/
# 3. Web-first assertion with no await: floats, never affects the result
grep -rnE "^[[:space:]]*expect\(.+\)\.to(BeVisible|BeHidden|BeEnabled|HaveText|HaveURL|HaveTitle|HaveCount)\(" e2e/
# 4. State reads whose results go nowhere
grep -rnE "^[[:space:]]*(await )?[A-Za-z_.]+\.(isVisible|isEnabled|isDisabled|isHidden)\(\)[[:space:]]*;?[[:space:]]*$" e2e/
Enter fullscreen mode Exit fullscreen mode
Every confirmed hit on #1 or #2 is a test, or a whole suite, that currently proves nothing.
First attempt: just lint it
My first instinct was that this is a lint problem, and some of it genuinely is. expect(locator).toBeTruthy() is a syntactic shape. A leaked .only is about as mechanical as it gets. So I built that layer and published it as two ESLint plugins, eslint-plugin-playwright-silent-pass and eslint-plugin-cypress-silent-pass, which flag the Carbon and Ghost class of bug on every future PR. (The install one-liner is at the end. The greps above are the no-install version.)
Then I hit the wall. Consider a test that clicks Delete, waits for the modal to close, and ends. Every line is idiomatic. Every assertion that exists is awaited and correct. The problem is the assertion that does not exist: nothing verifies the entity is gone. No AST rule can flag that, because there is no bad node to point at; the defect is the absence of a node. Same for a test named "accepts terms of use" that only asserts the pre-accept state, or an API check like expect([200, 401, 403]).toContain(status) that waves a broken auth endpoint through. Deciding those cases means reading the test the way a reviewer does: what does this test claim to prove, and does the code prove it?
That is a semantic judgment, and it is the kind of work an LLM can do well, provided you constrain it hard.
Measuring it instead of asserting it
"An LLM reviews your tests" is an easy claim to make badly. So before writing this post I benchmarked the approach and published the per-case evidence, so the numbers can be re-derived (methodology and results).
The setup: 100 open-source PRs across 77 distinct repositories, each touching Playwright or Cypress spec files, each already reviewed by one of eight AI PR review bots (CodeRabbit, Copilot, Codex, Sourcery, Gemini, Cursor, Ellipsis, Greptile). An LLM judge read every spec file, established a ground-truth set of 110 genuine test-trust issues, then scored three tools against it.
| Tool | Real issues caught | False positives | Unique catches |
|---|---|---|---|
| e2e-reviewer | 78 / 110 (71%) | 0 | 47 |
| lint (eslint-plugin-playwright / -cypress) | 45 / 110 (41%) | 0 | — |
| AI PR reviewers (inline spec comments) | 10 / 110 (9%) | 72 noise comments | 4 |
About that zero in the false-positive column: it means the judge ruled none of my reviewer's reported findings spurious, and you should still treat the cell with suspicion, for the second caveat below. Per PR, on the 33 PRs that contained a real issue, the win count was e2e-reviewer 19, lint-sufficient 11, AI reviewer 2, one tie.
The caveats, because the numbers are only as good as their weaknesses:
- The AI-reviewer figure is under-measured. The bots review the whole PR with split attention, and only their inline spec-file comments were counted, capped at six per PR. Much of their "noise" is legitimate feedback that is simply off-target for this ground truth. This measures a specialist against generalists on the specialist's home turf, nothing more.
- The judge and my reviewer share a model family, which can inflate recall and deflate false positives. As a pressure test, the 15 contestable unique catches were re-judged by a cross-model judge (OpenAI's gpt-5.5 via Codex), which upheld 13 of 15; the two disagreements were definitional edge cases, not overturned defects.
- The ground truth is LLM-established over a single 100-PR snapshot. Treat the exact figures as indicative rather than definitive.
Two results cut against my own product, and I want them in the post. First: on the purely mechanical layer, my scanner added net-new findings over standard lint in only 8 of 100 PRs. Run lint first. I mean that. All 47 unique catches came from the semantic layer; the wall was real, and it is where the value lives. Second: the benchmark shrank the tool. Three candidate rules mined from bot comments (custom sleep helpers, external-URL navigation, hardcoded localhost URLs) were rejected after measuring them across the 77 repos, either near-zero prevalence or close to 100% false-positive once project config was considered. The tool got smaller because of this data.
The findings also hold up in front of maintainers who have no reason to humor me. As of this writing, 12 upstream PRs fixing silent-pass tests have been reviewed and merged: Storybook (90k stars), code-server (78k), Strapi (72k), Ghost (54k), Cal.com (45k), Bruno (45k), Qwik (22k), SvelteKit (20k), Element Web (13k), Carbon Design System (9k), MUI X (5.8k), and module-federation/core (2.6k), with more in review (case studies). Every one of them fixed a test that was passing in CI while proving too little.
One more thing, because "AI-assisted PRs" have earned a reputation in 2026: these were not sprayed. Each fix was prepared against the target repo's contributing conventions and verified before submission, and the full tally is public — 12 merged, 8 in review. The whole pipeline, including what got dropped and why, lives in the roadmap.
What I built
e2e-skills is four Agent Skills for Claude Code, Codex, and other skill-compatible agents, built around a taxonomy of 24 E2E anti-patterns graded P0/P1/P2. P0 means silent always-pass, the tests that cannot fail. P1 means poor diagnostics when they do fail. P2 is maintenance debt.
The centerpiece, e2e-reviewer, runs in two phases. A deterministic scanner (plain bash, regex plus ast-grep, runnable standalone without any LLM) generates candidates; a semantic review phase then verifies each candidate in context and hunts for the absence-shaped defects the scanner cannot see, like the delete test that never checks deletion. That verification step is why the false-positive column reads zero: the LLM's job includes killing bad candidates, not only finding new ones. Alongside it, three skills close the loop. playwright-test-generator writes new coverage held to the same 24-pattern standard — on first run it scaffolds your project's conventions (an AGENTS.md E2E section plus a seed spec) and finishes with a YAGNI audit, so it does not ship a Page Object nobody calls. playwright-debugger and cypress-debugger start from the report files you already have (playwright-report/, mochawesome or JUnit) and classify each failure into one of 15 root-cause categories — flaky timing, selector drift, environment mismatch — each with a concrete fix; hand them a GitHub Actions run and they fetch the artifact themselves. Playwright and Cypress only, deliberately. For anything else, the skills say "out of scope" instead of guessing.
If you take one action from this post, run the greps above. If you want the mechanical patterns caught on every PR from now on, the zero-commitment entry is the lint layer:
npm i -D eslint-plugin-playwright-silent-pass # or eslint-plugin-cypress-silent-pass
Enter fullscreen mode Exit fullscreen mode
The skills, the standalone scanner, and the full benchmark evidence are at github.com/voidmatcha/e2e-skills. Both lint plugins are open source too: playwright · cypress.
The habit transfers even if you never install anything. Next time a suite goes green, ask it: if the feature broke tonight, which of you would actually fail?
