Get total files size from a file containing a file list

Question

I have a file containing a list of files that I would like to know the total files size. Is there a command to do so?

My OS is a very basic linux (Qnap TS-410).

EDIT:

A few lines from the file:

/share/archive/Bailey Test/BD006/0.tga
/share/archive/Bailey/BD007/1 version 1.tga
/share/archive/Bailey 2/BD007/example.tga

Answer 1

I believe something like this would work in busybox:

du `cat filelist.txt` | awk '{i+=$1} END {print i}'

I don't have the same environment as you, but if you encounter issues with spaces in filenames something like this would work too:

cat filelist.txt | while read file;do
  du "$file"
done | awk '{i+=$1} END {print i}'

Edit 1:
@stew is right in his post below, du shows the disk usage and not the exact filesize. To change the behavior busybox uses the -a flag, so try: du -a "$file" for exact filesize and compare the output/behavior.

Answer 2

du -c `cat filelist.txt` | tail -1 | cut -f 1

-c adds line "total size";
tail -1 takes last line (with total size);
cut -f 1 cuts out word "total".

Answer 3

I don't know if your linux tools are capable of this, but:

cat /tmp/filelist.txt  |xargs -d \\n du -c

Do, the xargs will set the delimiter to be a newline character, and du will produce a grand total for you.

Looking at http://busybox.net/downloads/BusyBox.html it seems that "busybox du" will support the grand total option, but the "busybox xargs" will not support custom delimiters.

Again, I'm not sure of your toolset.

Answer 4

while read filename ;  do stat -c '%s' $filename ; done < filelist.txt | awk '{total+=$1} END {print total}'

This is similar to Mattias Ahnberg's solution. Using "read" gets around problems with filenames/directories with spaces. I use stat instead of du to get the filesize. du is getting the amount of room it is using on disk instead of the filesize, which might be different. Depending on your filesystem, a 1 byte file will still occupy 4k on disk (or whatever the blocksize is). So for a 1 byte file, stat says 1 byte and du says 4k.

Answer 5

Here's another solution to the problem:

cat filelist.txt | tr '\n' '\0' | wc -c --files0-from=-

Answer 6

What about this other option:

tr \\n \\0 < filelist.txt | du -ch --files0-from=-

You can tail -1 the output to have only the total and/or use du -b instead of du -h to get the output in bytes.

Answer 7

Try something like this:

$ cat filelist.txt | xargs ls -l | awk '{x+=$5} END {print "total bytes: " x}' 

To deal properly with spaces in paths:

$ find /path/to/files -type f -print0 | xargs -0 ls -l | awk '{x+=$5} END {print "total bytes: " x}' 

Answer 8

This command checks the size of each file in the list and outputs the total in human readable format.

EDIT 07-02-2023: See the commands below. This (original) suggestion of mine parses the output from ls, which is prone to cause problems.

while read line; do ls -l "$line"; done < filelist.txt | awk '{total+=$5} END {print total}' | awk '{ split( "bytes KB MB GB TB PB" , v ); s=1; while( $1>1024 ){ $1/=1024; s++ } printf "Total: %.2f %s", $1, v[s] }'

It uses only ls, awk and printf, which are likely to exist on the most basic of systems. It avoids the need to use xargs as the read command takes its input one line at a time from the list of files.

It works as follows. It reads each line of the file containing the list (filelist.txt in my example here) and pipes the output into awk, which reads only field 5 ($5) and keeps adding to the total for each file. The ls command with the -l switch provides us with the file size in bytes, so this is fed into a second awk which is a function to repeatedly divide the bytes by 1024 to convert it to KB, MB, GB, TB and even PB if you have that much disk space! It will print whatever is appropriate, ie KB, GB etc to 2 decimal places. If you want 3 decimal places then change the 2 to a 3 in the printf command at the end, viz

printf "Total: %.3f %s", $1, v[s]...

Asked almost 11 years ago, last activity 6 years ago! Well, if anyone finds this, it does meet the OPs requirements and maybe it will help someone.

EDIT 07-02-2023:

As pointed out by @doneal24, we should not be parsing the output from ls. So we can simply read each line of the list of files, filelist.txt, and run du -b against each one. We then use awk to count the total bytes, and then again to convert it into human readable format. We do not need to pipe through xargs as the structure of the commands is that the filenames are presented singularly from the read command, which takes its input from the list one at a time.

while read line; do du -b "$line"; done < filelist.txt | awk '{i+=$1} END {print i}' | awk '{ split( "bytes KB MB GB TB PB" , v ); s=1; while( $1>1024 ){ $1/=1024; s++ } printf "Total: %.2f %s", $1, v[s] }'

I believe this is the answer the OP wanted - it provides the total size of the files in the list in human readable format.

Once again, thanks to @doneal24 for the input. It proves how, even answering a post over 10 years old, we continue to learn and help others. Teamwork!

Answer 9

cat docs.txt | xargs -d \\n du -sk | awk '{total+=$1} END{print total}'