1

When I include the NULL character (\x00) in a regex character range in BSD grep, the result is unexpected: no characters match. Why is this happening?

Here is an example:

$ echo 'ABCabc<>/ă' | grep -o [$'\x00'-$'\x7f']

Here I expect all characters up until the last one to match, however the result is no output (no matches).

Alternatively, when I start the character range from \x01, it works as expected:

$ echo 'ABCabc<>/ă' | grep -o [$'\x01'-$'\x7f']
A
B
C
a
b
c
<
>
/

Also, here are my grep and BASH versions:

$ grep --version
grep (BSD grep) 2.5.1-FreeBSD

$ echo $BASH_VERSION
3.2.57(1)-release
Lucas Leblow
  • 45
  • 7

2 Answers2

3

On BSD grep, you may be able to use this:

LC_ALL=C grep -o '[[:print:][:cntrl:]]' <<< 'ABCabc<>/ă'

A
B
C
a
b
c
<
>
/

Or you can just install gnu grep using home brew package and run:

grep -oP '[[:ascii:]]' <<< 'ABCabc<>/ă'
Arvind Kumar Avinash
  • 71,965
  • 6
  • 74
  • 110
anubhava
  • 761,203
  • 64
  • 569
  • 643
2

Noting that $'...' is a shell quoting construct, this,

$ echo 'ABCabc<>/ă' | grep -o [$'\x00'-$'\x7f']

would try to pass a literal NUL character as part of the command line argument to grep. That's impossible to do in any Unix-like system, as the command line arguments are passed to the process as NUL-terminated strings. So in effect, grep sees just the arguments -o and [.

You would need to create some pattern that matches the NUL byte without including it literally. But I don't think grep supports the \000 or \x00 escapes itself. Perl does, though, so this prints the input line with the NUL:

$ printf 'foo\nbar\0\n' |perl -ne 'print if /\000/'
bar

As an aside, at least GNU grep doesn't seem to like that kind of a range expression, so if you were to use that, you'd to do something different. In the C locale, [[:cntrl:][:print:]]' might perhaps work to match the characters from \x01 to \x7f, but I didn't check comprehensively. The manual for grep has some descriptions of the classes.

Note also that [$'\x00'-$'\x7f'] has an unquoted pair of [ and ] and so is a shell glob. This isn't related to the NUL byte, but if you had files that match the glob (any one-letter names, if the glob works on your system -- it doesn't on my Linux), or had failglob or nullglob set, it would probably give results you didn't want. Instead, quote the brackets too: $'[\x00-\x7f]'.

ilkkachu
  • 6,221
  • 16
  • 30
  • Are `grep ['a'-'b']` and `grep '[a-b]'` the same too? – oguz ismail Mar 02 '21 at 05:15
  • @oguzismail, other than the first being a glob, aren't they? – ilkkachu Mar 02 '21 at 09:32
  • That the first is a glob is not a negligible difference. They aren't the same. – oguz ismail Mar 02 '21 at 09:48
  • @oguzismail, for the purposes of this issue, it is a negligible difference. – ilkkachu Mar 02 '21 at 09:50
  • 1
    Misleading anyway. `['a'-'b']` may expand to empty string (`nullglob`), `a`, `b`, both `a` and `b`, or `[a-b]`, or may fail the command (`failglob`), while `'[a-b]'` will always be `[a-b]`. – oguz ismail Mar 02 '21 at 09:53
  • 1
    @oguzismail, yes, actually you're right in that the glob there in the question would match any one-letter name, assuming it works at all. I tend to, on purpose, ignore `failglob`, `nullglob` and the chance of matching strange filenames so as to keep to the point, but one-letter filenames are a more likely problem. Call it lazyness if you will, but the disclaimer about all the quirks of the shell language is sadly rather long to include in every post... – ilkkachu Mar 02 '21 at 10:40