Optimization: Python, Perl, and a C Suffix Tree Library

Question

I've got about 3,500 files that consist of single line character strings. The files vary in size (from about 200b to 1mb). I'm trying to compare each file with each other file and find a common subsequence of length 20 characters between two files. Note that the subsequence is only common between two files during each comparison, and not common among all files.

I've stuggled with this problem a bit, and since I'm not an expert, I've ended up with a bit of an ad-hoc solution. I use itertools.combinations to build a list in Python that ends up with around 6,239,278 combinations. I then pass the files two at a time to a Perl script that acts a wrapper for a suffix tree library written in C called libstree. I've tried to avoid this type of solution but the only comparable C suffix tree wrapper in Python suffers from a memory leak.

So here's my problem. I've timed it, and on my machine, the solution processes about 500 comparisons in 25 seconds. So that means, it'll take around 3 days of continuous processing to complete the task. And then I have to do it all again to look at say 25 characters instead of 20. Please note that I'm way out of my comfort zone and not a very good programmer, so I'm sure there is a much more elegant way to do this. I thought I'd ask it here and produce my code to see if anyone has any suggestion as to how I could complete this task faster.

Python code:

from itertools import combinations
import glob, subprocess

glist = glob.glob("Data/*.g")
i = 0

for a,b in combinations(glist, 2):
    i += 1
    p = subprocess.Popen(["perl", "suffix_tree.pl", a, b, "20"], shell=False, stdout=subprocess.PIPE)
    p = p.stdout.read()
    a = a.split("/")
    b = b.split("/")
    a = a[1].split(".")
    b = b[1].split(".")
    print str(i) + ":" + str(a[0]) + " --- " + str(b[0])
    if p != "" and len(p) == 20:
        with open("tmp.list", "a") as openf:
            openf.write(a[0] + " " + b[0] + "\n")

Perl code:

use strict;
use Tree::Suffix;

open FILE, "<$ARGV[0]";
my $a = do { local $/; <FILE> };

open FILE, "<$ARGV[1]";
my $b = do { local $/; <FILE> };

my @g = ($a,$b);

my $st  = Tree::Suffix->new(@g);
my ($c) = $st->lcs($ARGV[2],-1);

print "$c";

You don't need to find it for 20, then for 25. Just start by finding the LCS and saving it. Then you can simply check the length of the stored value for 20, 25, etc. — ikegami, Jul 21 '12 at 04:52
Alternatively, you could modify the LCS algorithm to exit as soon as an CS of the specified length is found. Easy to do. Both of these solutions only work if the you're ok with "finding a common subsequence 20 chars long" instead of "finding that the LCS is exactly 20 chars long". — ikegami, Jul 21 '12 at 05:22
25 s / 500 comp = 0.05 s/comp. 50ms to find an LCS doesn't sounds reasonable. btw, your math is a bit off. 3,500 * 3,499 comp * 0.05 s/comp = 612,325 s = 7.1 days, not 3 days. — ikegami, Jul 21 '12 at 05:22
you can write a suffix tree library in Python and then use Cython to optimize it — Ramchandra Apte, Jul 21 '12 at 13:06
An alternative I thought about was maybe mounting a ram drive (tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0) and then reading/writing from there. I tried it and the improvement doesn't seem amazing. I'll try Cython and see if I can do anything, but the answer below shows a Perl implementation that seems promising. @ikegami — mstcamus, Jul 22 '12 at 09:01

score 4 · Answer 1 · answered Jul 21 '12 at 16:42

Rather than writing Python to call Perl to call C, I'm sure you would be better off dropping the Python code and writing it all in Perl.

If your files are certain to contain exactly one line then you can read the pairs more simply by writing just

my @g = <>;

I believe the program below performs the same function as your Python and Perl code combined, but I cannot test it as I am unable to install libstree at present.

But as ikegami has pointed out, it would be far better to calculate and store the longest common subsequence for each pair of files and put them into categories afterwards. I won't go on to code this as I don't know what information you need - whether it is just subsequence length or if you need the characters and/or the position of the subsequences as well.

use strict;
use warnings;

use Math::Combinatorics;
use Tree::Suffix;

my @glist = glob "Data/*.g";
my $iterator = Math::Combinatorics->new(count => 2, data => \@glist);

open my $fh, '>', 'tmp.list' or die $!;

my $n = 0;
while (my @pair = $iterator->next_combination) {
  $n++;
  @ARGV = @pair;
  my @g = <>;
  my $tree  = Tree::Suffix->new(@g);
  my $lcs = $tree->lcs;
  @pair = map m|/(.+?)\.|, @pair;
  print "$n: $pair[0] --- $pair[1]\n";
  print $fh, "@pair\n" if $lcs and length $lcs >= 20;
}

Optimization: Python, Perl, and a C Suffix Tree Library

1 Answers1