18

How to check if a large file contains only zero bytes ('\0') in Linux using a shell command? I can write a small program for this but this seems to be an overkill.

vitaut
  • 49,672
  • 25
  • 199
  • 336

6 Answers6

19

If you're using bash, you can use read -n 1 to exit early if a non-NUL character has been found:

<your_file tr -d '\0' | read -n 1 || echo "All zeroes."

where you substitute the actual filename for your_file.

tzot
  • 92,761
  • 29
  • 141
  • 204
user2719058
  • 2,183
  • 11
  • 13
  • 2
    It's worth noting that `file` is the filename here, not the command `file`. – Sam Finnigan Sep 27 '15 at 11:13
  • 1
    I'd add: for people like me who don't know all bash syntax and don't like "cryptic" notations (this is of course subjective), this is equivalent to : `cat you_file | tr -d '\0' | read -n 1 || echo "All zeroes."` – Bruno Duyé Dec 22 '17 at 23:47
  • Also note, that you can drop the 'useless cat' by doing ```sh tr -d '\0' < [your_file] | read -n 1 || echo "All zeroes." ``` Though I'd favor a more posix way by dropping the -n 1 and using `wc` instead, – oliver Jun 10 '20 at 19:24
  • And since I was to slow to edit, here would be my suggestion: ```if [ "$(tr -d '\0' < [your_file] | wc -c)" -eq 0 ]; then echo "All zeroes." fi ``` where 'tr' deletes any character in the source file (`\0` in this case) and thus, the count should be 0. And sorry for the formatting, but < code > nor ` ` ` seem to work today? – oliver Jun 10 '20 at 19:33
15

The "file" /dev/zero returns a sequence of zero bytes on read, so a cmp file /dev/zero should give essentially what you want (reporting the first different byte just beyond the length of file).

vonbrand
  • 11,412
  • 8
  • 32
  • 52
  • 4
    Good answer. Actually one can tell `cmd` how many bytes to compare, so `cmp -n $(stat -c %s $FILE) $FILE /dev/zero` works like a charm. – Guido Feb 11 '19 at 12:20
  • this is better than the accepted answer from a not-reading-a-byte-at-a-time point of view. But if you're writing in bash, you probably don't care about that. – stu Jan 21 '20 at 16:19
  • 1
    @stu, if your file is a few megabytes long, it sure makes a difference. – vonbrand Jan 21 '20 at 16:53
  • yeah, I was more thinking, if your file is megabytes long and you're going to be doing it a lot, you wouldn't be using a scripting language – stu Jan 24 '20 at 02:56
  • 1
    @stu if you do it a few times, your time is much more valuable than the computer's – vonbrand Feb 03 '20 at 17:14
  • I spend a lot of the time the computer is doing something, doing something else, like sleeping. You can pay more for faster processors, or you can just schedule your time more wisely. – stu Feb 04 '20 at 23:48
  • 1
    @stu why not? Compiled, bummed for efficiency code for repetitive, heavy lifting; scripting for flexibility/choreography. – vonbrand Feb 25 '20 at 22:46
  • depends on the goal. if I'm writing something important for production that I don't want to get a call at 3am about, I'm going to put in lots of error checking and logging and redundancy and retry logic. Certainly you can do that in bash, but I'd prefer to do it in a language that already has logging libraries. that's more the angle I was getting at. In a bash script, I run it to make it go, and I don't care if it's a bit slow. If it needs to go faster, it's likely a production problem. – stu Feb 25 '20 at 23:53
  • I found this answer trying to check if a region of a file was all zeroes. Thanks for the help: `cmp --ignore-initial=$offset:0 --bytes=$size $file /dev/zero` appears to work just fine. Where `file` is the file I am checking, at an offset of `offset` bytes into the file, and the region is `size` bytes in length. The `:0` indicates not to skip initial bytes in `/dev/zero`. – neuralmer Aug 12 '22 at 18:25
9

If you have Bash,

cmp file <(tr -dc '\000' <file)

If you don't have Bash, the following should be POSIX (but I guess there may be legacy versions of cmp which are not comfortable with reading standard input):

tr -dc '\000' <file | cmp - file

Perhaps more economically, assuming your grep can read arbitrary binary data,

tr -d '\000' <file | grep -q -m 1 ^ || echo All zeros

I suppose you could tweak the last example even further with a dd pipe to truncate any output from tr after one block of data (in case there are very long sequences without newlines), or even down to one byte. Or maybe just force there to be newlines.

tr -d '\000' <file | tr -c '\000' '\n' | grep -q -m 1 ^ || echo All zeros
tripleee
  • 175,061
  • 34
  • 275
  • 318
5

It won't win a prize for elegance, but:

xxd -p file | grep -qEv '^(00)*$'

xxd -p prints a file in the following way:

23696e636c756465203c6572726e6f2e683e0a23696e636c756465203c73
7464696f2e683e0a23696e636c756465203c7374646c69622e683e0a2369
6e636c756465203c737472696e672e683e0a0a766f696420757361676528
63686172202a70726f676e616d65290a7b0a09667072696e746628737464
6572722c202255736167653a202573203c

So we grep to see if there is a line that is not made completely out of 0's, which means there is a char different to '\0' in the file. If not, the file is made completely out of zero-chars.

(The return code signals which one happened, I assumed you wanted it for a script. If not, tell me and I'll write something else)

EDIT: added -E for grouping and -q to discard output.

Guido
  • 2,571
  • 25
  • 37
  • 1
    The grouping with parentheses requires `grep -E` at least for me (but is obviously easy enough to fix). – tripleee Nov 26 '13 at 19:12
  • For some reason it worked for me without -E. I had it on and dropped it by accident. Edited to add it. Thanks! – Guido Nov 26 '13 at 19:14
  • It may not win a prize for elegance, but quite possibly one for speed. – Siguza Nov 30 '18 at 18:54
  • Also, this one wins (slightly modified) if files with runs of zeroes need to be identified. E.g. as read by 'cpio' from failing media. – yury10578 Feb 16 '22 at 15:15
0

Straightforward:

if [ -n $(tr -d '\0000' < file | head -c 1) ]; then
   echo a nonzero byte
fi

The tr -d removes all null bytes. If there are any left, the if [ -n sees a nonempty string.

arnt
  • 8,949
  • 5
  • 24
  • 32
-3

Completely changed my answer based on the reply here

Try

perl -0777ne'print /^\x00+$/ ? "yes" : "no"' file
Community
  • 1
  • 1
CodeBlue
  • 14,631
  • 33
  • 94
  • 132
  • How do I send the file contents to perl via command line in that case? I checked with empty file and it doesn't produce "yes". – CodeBlue Nov 26 '13 at 18:45
  • Sorry, empty files are fine but a file with one non-zero byte will produce "yes". – tripleee Nov 26 '13 at 18:47
  • Perl, like most every Unix file handling utility, can read a file without the help of `cat` (with `tr` perhaps being the one notable exception). `perl -ne 'print if s/^0+$/yes/' file` – tripleee Nov 26 '13 at 18:48
  • Edited script to include "+" instead of * – CodeBlue Nov 26 '13 at 18:49
  • It will still print something if there are characers which are not zeros. – tripleee Nov 26 '13 at 18:50
  • You are completely misunderstanding the OP. The question asks how to check if a file consists entirely of null characters (ASCII `0x00`) while your solution looks for literal zeros (ASCII `0x30`). @tripleee already pointed this out in an [earlier comment](http://stackoverflow.com/questions/20224822/how-to-check-if-a-file-contains-only-zeros-in-a-linux-shell/20225286#comment30161126_20225286) but you seem to have ignored that. – ThisSuitIsBlackNot Nov 26 '13 at 22:12
  • Oh, I completely missed that. :( Edited the answer – CodeBlue Nov 26 '13 at 22:15
  • 2
    Your new answer will read the entire file into memory, which is unattractive if the input file could be multiple gigabytes. – tripleee Nov 26 '13 at 22:19