Merging directories and joining inner files with bash

Question

I have bunch of directories following this structure:

\directories\
 |---\directory1\
 |      |---file1.txt
 |      |---file2.txt
 |
 |---\directory2\
 |      |---file1.txt
 |      |---file3.txt
 |
 |---\directory3\
 |      |---file4.txt
 |      |---file2.txt

I want to merge the directories and files so it ends up like this:

\directories\
 |---\directory1\
 |      |---file1.txt
 |      |---file2.txt
 |
 |---\directory2\
 |      |---file1.txt
 |      |---file3.txt
 |
 |---\directory3\
 |      |---file4.txt
 |      |---file2.txt
 |
 |---\mergeddata\
 |      |---file1.txt (from dir1 and dir2)
 |      |---file2.txt (from dir1 and dir4)
 |      |---file3.txt (from dir2)
 |      |---file4.txt (from dir3)

I am awful with bash and been trying quite a few things but... not getting any good results.

Looking forward to some help!

score 3 · Accepted Answer · answered Aug 06 '13 at 13:57

3

You didn't say how you want the files merged, or in what order. I will guess "concatenated, with directory1 appearing before diretory2, and directory2 before directory3"?

The following script shows a straightforward way to do this, without relying on fancy substitutions:

cd directories
mkdir mergeddata
for I in directory1 directory2 directory3 ; do   # replace with your actual directory list
    for F in "$I"/* ; do
        B=$(basename "$F")
        cat "$F" >> "mergeddata/$B"
    done
done

Edit: I added some quotes, in case any of your filenames end up with space characters or other inconvenient white space.

answered Aug 06 '13 at 13:57

Joe Z

17,413
3
28
39

Why do you use `cat` instead of `cp` to copy the file content? – Gearoid Murphy Aug 06 '13 at 14:22
1

@GearoidMurphy: Because the original question asked to merge files with like names. Notice that I'm also using the `>>` append operator to append to the target file as opposed to overwriting it. – Joe Z Aug 06 '13 at 14:36
+1 Useful use of cat! You can use `for I in */; do` to iterated over the subdirectories of "directories" – glenn jackman Aug 06 '13 at 15:06
@glennjackman: Doing `for I in */` would work if the target directory were not in the current directory. But in this example, `mergeddata` is in the same directory as `directory1`, `directory2` and `directory3`. The `for` loop would do surprising things once `I` becomes `mergeddata`. – Joe Z Aug 08 '13 at 14:57
In that case, since we're tagged [tag:bash], `shopt -s extglob; for I in !(mergedata)/` – glenn jackman Aug 08 '13 at 14:59

Merging directories and joining inner files with bash

1 Answers1