# A failure ceiling that only goes down

> Instead of tolerating a red suite or freezing work to fix everything at once, commit a failure ceiling that is calibrated only from completed CI runs, can only decrease, and refuses to grade stale or partial evidence. The ceiling stops new debt on day one; when the burn-down reaches zero, the same file becomes the permanent zero-failure enforcement.

URL: https://biloh.com.au/docs/engineering-notes/a-failure-ceiling-that-only-goes-down
Category: Engineering notes | Audience: builder | Updated: 2026-08-28

## The problem

A large suite with hundreds of known failures gives you the worst of both worlds: CI is always red, so a *new* failure is invisible, and "fix everything first" is a months-long freeze nobody will fund. The usual escape hatches — skip the failures, or stop looking — both destroy the suite's value.

## The mechanism

Commit a small JSON file to the repo: a **failure ratchet**.

```json
{
  "maxFailedTests": 118,
  "maxPendingTests": 0,
  "maxSilentlyDeadFiles": 0,
  "minTotalTests": 17000,
  "measuredFrom": { "source": "GitHub Actions", "runNumber": 65, "...": "..." }
}
```

A CI step compares the run's measured counts against the ceilings and fails the workflow when any is exceeded. The rules that make it trustworthy:

**The ceiling only decreases.** Every calibration lowers it to a measured count, never raises it. A structural test enforces this: it refuses any ceiling looser than the recorded measurement, so "just bump the number" is not a commit anyone can make.

**Calibration comes from CI, with provenance.** The `measuredFrom` block records the workflow, run number, run id, and commit. The same structural test refuses a measurement taken anywhere but CI. Local runs steer the work; only CI runs move the ceiling — scheduled runs sample the same UTC hour, so counts stay comparable run to run.

**Stale evidence is refused.** The checker refuses to grade a results artifact that is older than a day or predates the current HEAD, with an explicit `STALE RESULTS` error. A green verdict computed from last week's run is worse than no verdict.

**Partial evidence is refused.** A single-file test run writes the same results artifact as a full run — fresh, but describing 30 tests instead of 19,000. The `minTotalTests` floor makes the checker refuse it (`PARTIAL RESULTS`) rather than grade it as an impossibly good full run. This hole was found in review before it shipped; it would have made every false green look measured.

**Dead files are counted separately.** A spec whose `beforeAll` hook throws runs zero tests and contributes zero failures — a failure ceiling cannot see it, even at zero. `maxSilentlyDeadFiles` counts files that report no tests, and it stays in the file forever for exactly that reason.

## Why not just fix everything?

The ratchet changes the economics. The moment it lands, **new failures are blocked** — the suite's guard value is restored immediately, at the current debt level. The burn-down then proceeds in ordinary work, each session lowering the ceiling to its new measured count. On this platform that trajectory ran 561 → 494 → 465 → 431 → 399 → 305 → 118 → 0 across seventeen sessions, with the ceiling locking in each session's progress so it could never be given back.

## What happens at zero

Nothing is replaced. At `maxFailedTests: 0`, the ratchet **is** the zero-failure enforcement: the first regression fails the workflow on the next run. The pending and dead-file ceilings and the total floor stay, because zero failures still cannot see a hook-death or a partial artifact. The file's own notes record that the numbers may never rise and where the full history lives.

## The one prerequisite

The counts must be honest before you calibrate. A ceiling calibrated from a suite full of vacuous checks locks in fiction — see [Five ways a test suite lies to you](/docs/engineering-notes/five-ways-a-test-suite-lies) for the patterns to purge first.

## Related

- [Five ways a test suite lies to you](/docs/engineering-notes/five-ways-a-test-suite-lies)
- [Freeze the baseline, then burn it down](/docs/engineering-notes/freeze-the-baseline-then-burn-it-down)
- [The gap between a green build and a live deploy](/docs/engineering-notes/green-build-vs-live-deploy)