1

Executive Summary

Is it standard behavior that shells skip over NUL bytes when doing process substitution?

For example, executing

printf '\0abc' | read value && echo $value

will yield abc. The NUL value is skipped, even though the hexdump of the printf output shows it's clearly being output.

My first thought was "word splitting". However, when using an actual process substitution

value=$(printf '\0abc')

the results are similar and = does not perform word splitting.

Long Story

While searching for the proper answer for this question, I realized that at least three of the shell implementation (ash, zsh, and bash) I am reasonably familiar with will ignore a NUL character when reading the value from process substitution into a variable.

The exact point in the pipeline when this happens seems to be different, but the result is consistently that a NUL byte gets dropped as if it was never there in the first place.

I have checked with some of the implementations, and well, this seems to be normal behavior.

ash will skip over '\0' on input, but it is not clear from the code if this is pure coincidence or intended behavior:

if (lastc != '\0') {
    [...]
}

The bash source code contains an explicit, albeit #ifdef'd warning telling us that it skipped a NUL value on process substitution:

#if 0
      internal_warning ("read_comsub: ignored null byte in input");
#endif

I'm not so sure about zsh's behaviour. It recognizes '\0'as a meta character (as defined by the internal imeta() function) and prepends a special Meta surrogate character and sets bit #5 on the input character, essentially unmetaing it, which makes also makes '\0' into a space ' ')

if (imeta(c)) {
    *ptr++ = Meta;
    c ^= 32;
    cnt++;
}

This seems to get stripped later because there is no evidence that value in the above printf command contains a meta character. Take this with a large helping of salt, since I'm not to familiar with zsh's internals. Also note the side effect free statements.

Note that zsh also allows you to include NUL (meta-escaped) in IFS (making it possible to e.g. word-split find -print0 without xargs -0). Thus printf '\0abc' | read value and value=$(printf '\0abc') should yield different results depending on the value of IFS (read does field splitting).

dhke
  • 15,008
  • 2
  • 39
  • 56
  • Interesting! but I think you've answered your own question :) – davmac Sep 22 '15 at 16:29
  • @davmac not necessarily. It could be that this behavior was simply "there" in the initial Bourne shell. With bash it's more or less clear that this is deliberate from the warning, but I'm not so sure, if this was *intended* or just *it's been that way initially*. But nonetheless, it's indeed interesting even if unanswered. – dhke Sep 22 '15 at 16:41
  • 1
    POSIX shells use C strings. C strings can't contain NUL bytes. So, err, what do you expect? – Charles Duffy Sep 22 '15 at 16:45
  • 2
    BTW, you *can* represent a stream containing NUL bytes: Read into an array with NULs as separators. – Charles Duffy Sep 22 '15 at 16:46
  • Well, it's not _standardised_ in the sense that there's no documentation stating that it happens, at least not that I'm aware of and certainly not in the Bash manual. I suspect the NULs are stripped because the alternative is dealing with them properly, and that would take more effort :). – davmac Sep 22 '15 at 16:47
  • If "dealing with them properly" means having values you can't pass to C APIs, like those POSIX defines... – Charles Duffy Sep 22 '15 at 16:48
  • 2
    http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_03 says null bytes in command substitution output is undefined behavior. That's the closest spec language about this I can find offhand. – Etan Reisner Sep 22 '15 at 16:49
  • @CharlesDuffy it's perfectly possible to maintain a buffer containing NUL bytes and keep track of its size in an independent variable. (You can even pass this buffer to a call to POSIX functions like `write`). So "dealing with them properly" just means doing exactly this. – davmac Sep 22 '15 at 16:54
  • @davmac, obviously, but then what can you do with them? Shells are all about combining tools together; if you can't put a literal string in an argv list or pass it to a syscall, it's not good for much. – Charles Duffy Sep 22 '15 at 16:56
  • @davmac, ...I'd also argue that being able to store variables with NUL literals would make it much, much harder to write secure shell scripts, since part of their contents (only part!) would be discarded on being passed to UNIX syscalls (open, etc), so you'd have values which would compare differently but have the same effect in several uses. – Charles Duffy Sep 22 '15 at 16:58
  • @CharlesDuffy I'm not sure I see your point. You can still manipulate a string containing nuls in the same ways that you can manipulate a string not containing nuls. True, you would not be able to use such a string directly as an argument to another process but you could certainly use it as input for another process (so there's one thing that you can do with them). (And for the security aspect, it would be easy enough to just error out if a nul-containing string was used as a process argument). – davmac Sep 22 '15 at 16:59
  • @davmac, re: "would not be able to use such a string directly" -- and therein is the rub. If the implementation does the simple thing and just copies the string's contents in, then you get a string that's truncated early, and unexpected string truncation making comparisons invalid is an easy path to security bugs. From just this last week, see https://blog.perimeterx.com/bugzilla-cve-2015-4499/ – Charles Duffy Sep 22 '15 at 17:01
  • @davmac ...and, keep in mind, just argvs aren't enough; it would have to be tested for and managed when strings are passed to **any** syscall (`open`, for instance). That's a big code auditing and correctness effort (and nonzero performance impact) for a tiny corner case. – Charles Duffy Sep 22 '15 at 17:05
  • @CharlesDuffy right. Maybe we're arguing two different things: me that it _could_ be done, you that it _should not_ be done (which I tend to agree with). I doubt there are a lot of shell scripts that need to deal with nul bytes and I'd probably suggest that such things are better managed in another language. (OTOH I wouldn't be surprised if _some_ people have found out about this nul stripping the hard way). – davmac Sep 22 '15 at 17:09
  • *sigh* It was perfectly clear for me from the onset of the question that the environment cannot contain strings that extend past the first NUL. It should have been even clearer when noticing that I actually linked the sources. I am actually quite taken aback by the perceived compulsion to lecture about C strings (which is also why the answer is not accept yet, because I do not like to reward what I perceive as unwarranted snarkiness). The question was about if *ignoring* NULs during process substitution is defined behavior. It isn't. I am, however, quite sad on the course of this discussion. – dhke Sep 22 '15 at 17:10
  • Answers aren't just written for the person asking the question; they're written for the general audience. I don't intend to come off as condescending *to you*, but I do intend to ensure that all readers, not only yourself, have full context. – Charles Duffy Sep 22 '15 at 17:13
  • That doesn't change my perception that you assumed missing knowledge from the very first start and acted from that assumption. And I really mean that as constructive criticism. Ah, and to answer your first question: I did not expect anything (see the answer to the linked question, where I essentially state that I believe this to be undefined behavior). But my first guess was that `value` would be an empty string (terminated by the NUL in the input). I found it interesting, that ash and bash simply dropped NULs while zsh does meta-quote magick. – dhke Sep 22 '15 at 17:23
  • *nod*. I do recall being a bit surprised by bash's behavior (expecting truncation) on first encounter myself. As for zsh, on the other hand, it doesn't particularly try to comply with POSIX, so nothing would surprise me even if the standard _did_ specify a specific behavior. – Charles Duffy Sep 22 '15 at 17:31

1 Answers1

4

All extant POSIX shells use C strings (NUL-terminated), not Pascal strings (carrying their length as separate metadata, thus able to contain NULs). Thus, they can't possibly contain NULs in string contents. This was notably true of the Bourne Shell and ksh, both major influences to the POSIX sh standard.

The specification allows shells to behave in an implementation-defined manner here; without knowing the specific shell and release being targeted, I would not expect a specific behavior between terminating the stream returned at the first NUL and simply discarding NULs altogether. Quoting:

The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more characters at the end of the substitution. Embedded characters before the end of the output shall not be removed; however, they may be treated as field delimiters and eliminated during field splitting, depending on the value of IFS and quoting that is in effect. If the output contains any null bytes, the behavior is unspecified.

This isn't to say you can't read and produce streams containing NULs in widely-available shells! See the below, using process substitution (written for bash, but should work with ksh or zsh with minor changes if any):

# read content from stdin into array variable and a scalar variable "suffix"
array=( )
while IFS= read -r -d '' line; do
  array+=( "$line" )
done < <(process that generates NUL stream here)
suffix=$line # content after last NUL, if any

# emit recorded content
printf '%s\0' "${array[@]}"; printf '%s' "$suffix"
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441