Writing a script for large text file manipulation (iterative substitution of duplicated lines), weird bugs and very slow.

Question

I am trying to write a script which takes a directory containing text files (384 of them) and modifies duplicate lines that have a specific format in order to make them not duplicates.

In particular, I have files in which some lines begin with the '@' character and contain the substring 0:0. A subset of these lines are duplicated one or more times. For those that are duplicated, I'd like to replace 0:0 with i:0 where i starts at 1 and is incremented.

So far I've written a bash script that finds duplicated lines beginning with '@', writes them to a file, then reads them back and uses sed in a while loop to search and replace the first occurrence of the line to be replaced. This is it below:

#!/bin/bash                                                                                                                                      


fdir=$1"*"

#for each fastq file

for f in $fdir
do
    (

#find duplicated read names and write to file $f.txt

sort $f | uniq -d | grep ^@  > "$f".txt
#loop over each duplicated readname

    while read in; do
        rname=$in
        i=1

        #while this readname still exists in the file increment and replace

        while grep -q "$rname" $f; do
            replace=${rname/0:0/$i:0}
            sed -i.bu "0,/$rname/s/$rname/$replace/" "$f"
            let "i+=1"

        done

    done < "$f".txt



rm "$f".txt
rm "$f".bu

done

echo "done" >> progress.txt

)&

background=( $(jobs -p) )
if (( ${#background[@]} ==40)); then
wait -n
fi

done

The problem with it is that its impractically slow. I ran it on a 48 core computer for over 3 days and it hardly got through 30 files. It also seemed to have removed about 10 files and I'm not sure why.

My question is where are the bugs coming from and how can I do this more efficiently? I'm open to using other programming languages or changing my approach.

EDIT

Strangely the loop works fine on one file. Basically I ran

sort $f | uniq -d | grep ^@  > "$f".txt


while read in; do
    rname=$in
    i=1

    while grep -q "$rname" $f; do
        replace=${rname/0:0/$i:0}
        sed -i.bu "0,/$rname/s/$rname/$replace/" "$f"
        let "i+=1"

    done

done < "$f".txt

To give you an idea of what the files look like below are a few lines from one of them. The thing is that even though it works for the one file, it's slow. Like multiple hours for one file of 7.5 M. I'm wondering if there's a more practical approach.

With regard to the file deletions and other bugs I have no idea what was happening Maybe it was running into memory collisions or something when they were run in parallel?

Sample input:

@D00269:138:HJG2TADXX:2:1101:0:0 1:N:0:CCTAGAAT+ATTCCTCT
GATAAGGACGGCTGGTCCCTGTGGTACTCAGAGTATCGCTTCCCTGAAGA
+
CCCFFFFFHHFHHIIJJJJIIIJJIJIJIJJIIBFHIHIIJJJJJJIJIG
@D00269:138:HJG2TADXX:2:1101:0:0 1:N:0:CCTAGAAT+ATTCCTCT
CAAGTCGAACGGTAACAGGAAGAAGCTTGCTTCTTTGCTGACGAGTGGCG

Sample output:

@D00269:138:HJG2TADXX:2:1101:1:0 1:N:0:CCTAGAAT+ATTCCTCT
GATAAGGACGGCTGGTCCCTGTGGTACTCAGAGTATCGCTTCCCTGAAGA
+
CCCFFFFFHHFHHIIJJJJIIIJJIJIJIJJIIBFHIHIIJJJJJJIJIG
@D00269:138:HJG2TADXX:2:1101:2:0 1:N:0:CCTAGAAT+ATTCCTCT
CAAGTCGAACGGTAACAGGAAGAAGCTTGCTTCTTTGCTGACGAGTGGCG

Answer 1

Here's some code that produces the required output from your sample input.

Again, it is assumed that your input file is sorted by the first value (up to the first space character).

time awk '{
        #dbg if (dbg) print "#dbg:prev=" prev
        if (/^@/ && prev!=$1) {fixNum=0 ;if (dbg) print "prev!=$1=" prev "!=" $1}
        if (/^@/ && (prev==$1 || NR==1) ) {
                prev=$1
                n=split($1,tmpArr,":") ; n++
                #dbg if (dbg) print "tmpArr[6]="tmpArr[6] "\tfixNum="fixNum
                fixNum++;tmpArr[6]=fixNum;

                # magic to rebuild $1 here
                for (i=1;i<n;i++) {
                        tmpFix ? tmpFix=tmpFix":"tmpArr[i]"" : tmpFix=tmpArr[i]
                }
                $1=tmpFix ; $0=$0  
                print  $0
        }
        else { tmpFix=""; print $0 } 
        }' file > fixedFile

output

@D00269:138:HJG2TADXX:2:1101:1:0 1:N:0:CCTAGAAT+ATTCCTCT
GATAAGGACGGCTGGTCCCTGTGGTACTCAGAGTATCGCTTCCCTGAAGA
+
CCCFFFFFHHFHHIIJJJJIIIJJIJIJIJJIIBFHIHIIJJJJJJIJIG
@D00269:138:HJG2TADXX:2:1101:2:0 1:N:0:CCTAGAAT+ATTCCTCT
CAAGTCGAACGGTAACAGGAAGAAGCTTGCTTCTTTGCTGACGAGTGGCG

I've left a few of the #dbg:... statements in place (but they are now commented out) to show how you can run a small set of data as you have provided, and watch the values of variables change.

Assuming a non-csh, you should be able to copy/paste the code block into a terminal window cmd-line and replace file > fixFile at the end with your real file name and a new name for the fixed file. Recall that awk 'program' file > file (actually, any ...file>file) will truncate the existing file and then try to write, SO you can lose all the data of a file trying to use the same name.

There are probably some syntax improvements that will reduce the size of this code, and there might be 1 or 2 things that could be done that will make the code faster, but this should run very quickly. If not, please post the result of time command that should appear at the end of the run, i.e.

real    0m0.18s
user    0m0.03s
sys     0m0.06s

IHTH

Answer 2

#!/bin/bash                                                                     

i=4

sort $1 | uniq -d | grep ^@ > dups.txt

while read in; do

    if [ $((i%4))=0 ] && grep -q "$in" dups.txt; then
        x="$in"
        x=${x/"0:0 "/$i":0 "}
        echo "$x" >> $1"fixed.txt"

    else
        echo "$in" >> $1"fixed.txt"

    fi

    let "i+=1"
done < $1