# Five ways a test suite lies to you

> A green check is only evidence if the check can actually go red. Driving a ~20,000-test suite from 619 failures to a measured zero surfaced five distinct ways a check can pass without testing anything — each one found because a claim was verified against primary evidence instead of trusted.

URL: https://biloh.com.au/docs/engineering-notes/five-ways-a-test-suite-lies
Category: Engineering notes | Audience: builder | Updated: 2026-08-28

## The claim

A test that passes is making a claim: *the behaviour it asserts holds*. But a check can pass for a second, worse reason — because it never actually ran its assertion. During a drive that took this platform's suite from 619 failing tests to a measured zero, we found five separate shapes of that lie, some of them years old. Every one had been green the whole time.

## 1. The scanner that shells out to a tool that isn't there

Several structural checks ran `execSync("grep …")` inside a `try/catch`. On Linux CI they worked. On a Windows dev machine, `grep` is not on the PATH — the command throws, the catch swallows it, and the scanner reports **zero offenders, forever**. The check was not weak; it was *absent*, and indistinguishable from passing.

Fix at the cause: rewrite scanners in pure Node (walk the tree, read the files), and make the error path **throw**, never return an empty result. A scanner that cannot run must fail loudly, not pass quietly.

## 2. `|| true` that forgives more than flakiness

A shell step used grep with a BRE escape (`\(`) that the installed grep rejected as a syntax error — and the trailing `|| true`, added to tolerate "no matches", also converted *the tool erroring out* into success. The check passed on every run, including the ones where it crashed on line one.

`|| true` cannot distinguish "searched and found nothing" from "never searched". If a step can fail in two ways with different meanings, the exit codes must be handled separately.

## 3. `.todo` is not `.skip`

The suite had a structural guard asserting **no skipped tests**. It scanned for `.skip` — and `it.todo(...)` sailed straight through, because a todo is not a skip in the runner's accounting. Three placeholder tests sat invisible in a "no test left behind" suite.

If you enforce "nothing is silently disabled", enumerate every disabling mechanism your runner offers: `skip`, `todo`, `fails`, conditional skips, and empty bodies.

## 4. The spec file with nothing inside

A quarantined spec had been reduced, over successive edits, to a file that asserted nothing. It "passed" on every run — a file with zero failing assertions is green by definition. The fix was an integrity lock over the quarantine directory: every spec must contain real assertions, carry a header naming its owning work package, and the directory is capped so it cannot silently become a landfill.

## 5. The reporter that never loaded

The most instructive one. A speed-budget reporter was configured in the test config and had "run" on CI for months. Three layers deep, it never had:

1. the enforcement flag defaulted off;
2. CI's clean install meant the durations file it read never existed; and — decisively —
3. the CI workflow passed `--reporter=json` on the command line, and **a CLI `--reporter` flag *replaces* config-file reporters rather than adding to one**. The speed reporter was never even loaded.

The general trap: config composition rules differ between "merge" and "replace" per field, per tool. If a reporter, plugin, or hook matters, make CI name it **explicitly in the workflow**, and prove it ran by checking for its output artifact.

## The discipline that finds these

None of these were found by reading dashboards. Each was found by demanding the check be **demonstrated red** — deliberately planting an offender and watching the check catch it — before its green was trusted. A check you have never seen fail is a check you know nothing about.

The same discipline applies to time: a related class of flake, where a test seeds "today" from the runner's clock while the code buckets it in the tenant's timezone, is covered in [Whose "today" is it?](/docs/engineering-notes/whose-today-is-it) — and it too was closed with a detector that was verified red against a planted offender first.

## Related

- [The gap between a green build and a live deploy](/docs/engineering-notes/green-build-vs-live-deploy)
- [A handler nothing subscribes to does nothing](/docs/engineering-notes/wiring-an-event-to-its-handler)
- [A failure ceiling that only goes down](/docs/engineering-notes/a-failure-ceiling-that-only-goes-down)