merging big text files together

Question

I am merging many files (~1 gig each) into one file but the merged file is incomplete. when concatenating b to a, b gets concatenated somewhere in the middle rather than end. The command I am running are:

for f in $x/*/y/*.fastq; do

    fullpath=`echo $(readlink -f $f)`
    basename=`echo "${fullpath##*/}"`
    pathname=`echo "${fullpath%/*}"`
    name=`echo "$basename"|sed 's/-_-.*//'`

    cat $f>>$x/z/${name}.fastq

done

also, alternatively

names=$(cut -f 3 $B)
names=$(echo "${names[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' ')
for name in ${names[@]}; do
    cat $x/*/y/${name}-_-*.fastq>$x/z/${name}.fastq
done

after I inspect the file, the merged file has smaller size than original and also concatenated somewhere in middle.

Thanks

run your script with `bash -x` to see, what it really doing. — clt60, Aug 18 '14 at 18:46

score 0 · Answer 1 · answered Aug 18 '14 at 19:35

Try this one:

#!/bin/bash
shopt -s nullglob
for f in "$x"/*/y/*.fastq; do
    fullpath=$(readlink -f "$f")
    basename=${fullpath##*/}
    pathname=${fullpath%/*}
    name=${basename%%-_-*}
    dest=$x/z/$name.fastq
    echo "$f >> $dest"
    cat "$f" >> "$dest"
done

merging big text files together

1 Answers1