The claim
A test that passes is making a claim: the behaviour it asserts holds. But a check can pass for a second, worse reason — because it never actually ran its assertion. During a drive that took this platform's suite from 619 failing tests to a measured zero, we found five separate shapes of that lie, some of them years old. Every one had been green the whole time.
1. The scanner that shells out to a tool that isn't there
Several structural checks ran execSync("grep …") inside a try/catch. On Linux CI they worked. On a Windows dev machine, grep is not on the PATH — the command throws, the catch swallows it, and the scanner reports zero offenders, forever. The check was not weak; it was absent, and indistinguishable from passing.
Fix at the cause: rewrite scanners in pure Node (walk the tree, read the files), and make the error path throw, never return an empty result. A scanner that cannot run must fail loudly, not pass quietly.
2. || true that forgives more than flakiness
A shell step used grep with a BRE escape (\() that the installed grep rejected as a syntax error — and the trailing || true, added to tolerate "no matches", also converted the tool erroring out into success. The check passed on every run, including the ones where it crashed on line one.
|| true cannot distinguish "searched and found nothing" from "never searched". If a step can fail in two ways with different meanings, the exit codes must be handled separately.
3. .todo is not .skip
The suite had a structural guard asserting no skipped tests. It scanned for .skip — and it.todo(...) sailed straight through, because a todo is not a skip in the runner's accounting. Three placeholder tests sat invisible in a "no test left behind" suite.
If you enforce "nothing is silently disabled", enumerate every disabling mechanism your runner offers: skip, todo, fails, conditional skips, and empty bodies.
4. The spec file with nothing inside
A quarantined spec had been reduced, over successive edits, to a file that asserted nothing. It "passed" on every run — a file with zero failing assertions is green by definition. The fix was an integrity lock over the quarantine directory: every spec must contain real assertions, carry a header naming its owning work package, and the directory is capped so it cannot silently become a landfill.
5. The reporter that never loaded
The most instructive one. A speed-budget reporter was configured in the test config and had "run" on CI for months. Three layers deep, it never had:
- the enforcement flag defaulted off;
- CI's clean install meant the durations file it read never existed; and — decisively —
- the CI workflow passed
--reporter=jsonon the command line, and a CLI--reporterflag replaces config-file reporters rather than adding to one. The speed reporter was never even loaded.
The general trap: config composition rules differ between "merge" and "replace" per field, per tool. If a reporter, plugin, or hook matters, make CI name it explicitly in the workflow, and prove it ran by checking for its output artifact.
The discipline that finds these
None of these were found by reading dashboards. Each was found by demanding the check be demonstrated red — deliberately planting an offender and watching the check catch it — before its green was trusted. A check you have never seen fail is a check you know nothing about.
The same discipline applies to time: a related class of flake, where a test seeds "today" from the runner's clock while the code buckets it in the tenant's timezone, is covered in Whose "today" is it? — and it too was closed with a detector that was verified red against a planted offender first.