The problem
A large suite with hundreds of known failures gives you the worst of both worlds: CI is always red, so a new failure is invisible, and "fix everything first" is a months-long freeze nobody will fund. The usual escape hatches — skip the failures, or stop looking — both destroy the suite's value.
The mechanism
Commit a small JSON file to the repo: a failure ratchet.
{
"maxFailedTests": 118,
"maxPendingTests": 0,
"maxSilentlyDeadFiles": 0,
"minTotalTests": 17000,
"measuredFrom": { "source": "GitHub Actions", "runNumber": 65, "...": "..." }
}
A CI step compares the run's measured counts against the ceilings and fails the workflow when any is exceeded. The rules that make it trustworthy:
The ceiling only decreases. Every calibration lowers it to a measured count, never raises it. A structural test enforces this: it refuses any ceiling looser than the recorded measurement, so "just bump the number" is not a commit anyone can make.
Calibration comes from CI, with provenance. The measuredFrom block records the workflow, run number, run id, and commit. The same structural test refuses a measurement taken anywhere but CI. Local runs steer the work; only CI runs move the ceiling — scheduled runs sample the same UTC hour, so counts stay comparable run to run.
Stale evidence is refused. The checker refuses to grade a results artifact that is older than a day or predates the current HEAD, with an explicit STALE RESULTS error. A green verdict computed from last week's run is worse than no verdict.
Partial evidence is refused. A single-file test run writes the same results artifact as a full run — fresh, but describing 30 tests instead of 19,000. The minTotalTests floor makes the checker refuse it (PARTIAL RESULTS) rather than grade it as an impossibly good full run. This hole was found in review before it shipped; it would have made every false green look measured.
Dead files are counted separately. A spec whose beforeAll hook throws runs zero tests and contributes zero failures — a failure ceiling cannot see it, even at zero. maxSilentlyDeadFiles counts files that report no tests, and it stays in the file forever for exactly that reason.
Why not just fix everything?
The ratchet changes the economics. The moment it lands, new failures are blocked — the suite's guard value is restored immediately, at the current debt level. The burn-down then proceeds in ordinary work, each session lowering the ceiling to its new measured count. On this platform that trajectory ran 561 → 494 → 465 → 431 → 399 → 305 → 118 → 0 across seventeen sessions, with the ceiling locking in each session's progress so it could never be given back.
What happens at zero
Nothing is replaced. At maxFailedTests: 0, the ratchet is the zero-failure enforcement: the first regression fails the workflow on the next run. The pending and dead-file ceilings and the total floor stay, because zero failures still cannot see a hook-death or a partial artifact. The file's own notes record that the numbers may never rise and where the full history lives.
The one prerequisite
The counts must be honest before you calibrate. A ceiling calibrated from a suite full of vacuous checks locks in fiction — see Five ways a test suite lies to you for the patterns to purge first.