I have a directory with 27K files, I would like to tar them into multiple INDEPENDENT tar files, each will have 5000 files and the last one, will obviously have 2K (27K is not dividable by 5).
What is the fastest/easiest way to do this?
Asked
Active
Viewed 1,690 times
2 Answers
9
First create files with the filenames for each archive:
find <directory> | split -l 5000 - files.
Then create the tars:
for f in files.*; do tar -cf $f.tar --files-from $f; done
Untested but the basic idea should work.
Rob Wouters
- 1,927
- 1
- 11
- 4
-
1This is probably the quickest (in terms of typing) / most portable (should work on any unix - `find`, `split` and `for` loops in Bourne shells are everywhere) solution :) – voretaq7 Jan 26 '12 at 07:47
-
Use "find
-type f" so that the pathnames passed into tar don't include directories. Alternative: for small split sizes, you could could use xargs insteads of split and avoid all the temp files. – Liudvikas Bukys Jan 26 '12 at 14:25
0
#!/bin/bash
files=( dir/* )
n=1
for ((i = 0; i < ${#files[@]}; i+=5000)); do
IFS=$'\n' tar cvzf foo_$((n++)).tgz --files-from - <<<"${files[*]:i:5000}"
done
Explanation
Create an array called files that contains all contents of directory dir. Iterate over the array increasing the iterator 5000 at a time. Use the offset parameter expansion on the array to do a sliding window of up to 5000 items at a time "${files[@]:i:5000}" and pass that to tar and increment the file name by one $((n++)) on each call.
SiegeX
- 567
- 1
- 6
- 16
-
2Bash might throw an "argument list too long" at you if you do this. It's possible to pipe the filenames to tar if you use `--files-from -`. That might work better here. – Rob Wouters Jan 26 '12 at 06:37
-
@RobWouters Great idea. Did it in a slightly different way but this shouldn't cause any max arg errors. – SiegeX Jan 26 '12 at 06:59
-
1Aren't you passing all those arguments to `printf` now though? Same problem. – Rob Wouters Jan 26 '12 at 07:04
-