Optimization of the Burrows Wheeler transform

Question

Hello I am having some difficulty optimizing the burrows wheeler transform. I'm trying to transform text files, however transforming large text files like the bible take way too long.

Any idea on how to proceed?

public BurrowsWheelerTransformEncoder()
{

}

private String originalSuffix(int index, String string)
{
    String temp = (string.substring(index,string.length()) + string.substring(0,index));

    //this bit just 'compresses' each transformation of text by producing
    //a prefix, so 'abracadabra' just becomes 'abrac'
    //this is so minimal amount of memory is used when it is stored in an array

    return temp.substring(0,5)+
    //the last character of the transformation is kept
           temp.charAt(temp.length()-1);
}

private String compressedSuffix(String string)
{
    //this method just 'compresses' original piece of text by producing
    //a prefix, so 'abracadabra' just becomes 'abrac'
    //this is so comprisons won't take so long
    return string.substring(0,5)+string.charAt(string.length()-1);
}

public static void main(String args[]) throws Exception
{
    BurrowsWheelerTransformEncoder encoder = new BurrowsWheelerTransformEncoder();
    BufferedReader input = new BufferedReader(new FileReader("src/compressionalgorithm/texts/manifesto.txt"));

    String text = "";
    //the row in the sorted array where the original text can be found
    int originalRow = 0;
    //system time when program began
    long startTime = System.nanoTime();

    //get text from file
    while(input.ready())
    {
        text += input.readLine();
    }
    //create a new array to hold all transformations
    String[] textArray = new String[text.length()];
    int length = text.length();

    //get individual transformations and put in array
    for(int i = 0; i < text.length(); i++)
    {
        textArray[i] = encoder.originalSuffix(i,text);
        //for debugging large text files, prints progress after every 10k'th 
        //transformation
        if(i%10000==0)
        System.out.println(i+"/"+length);
    }
    //uses java's internal methods to sort the array, presumably 
    //the most efficient way to do the sort (for now)
    Arrays.sort(textArray);

    String compressedOriginalText = encoder.compressedSuffix(text);

    //print the results
    for(int i = 0; i < textArray.length; i++)
    {
        if(textArray[i].equals(compressedOriginalText))
        {
            originalRow = i;
        }
        if(i%100==0)
        {
            System.out.println();
        }
        System.out.print(textArray[i].charAt(textArray[i].length()-1));
    }
    System.out.println("\nThe original transformation of the text was found at row " + originalRow + " of the sorted array.");
    System.out.println("Time elapsed: " + (System.nanoTime() - startTime));
 }

score 3 · Answer 1 · edited Nov 08 '11 at 21:21

For the coding case, you don't need to actually build a string array - use an int (or long depending on your file size) array instead to store the index that your rotating string starts at.

Create an array initialized to [0 1 2 3 ... n]

sort the array with the following compareTo (assume compareTo() has access to the original string, original):

int compareTo(int a, int b){
    int compare, len = original.length();
    do{
        char _a = original.charAt(a), _b = original.charAt(b);
        compare = _a-_b;
        a++; b++;
        if(a>=len)a-=len;
        if(b>=len)b-=len;
    }while(compare==0);
    return compare;
}

note the index of "0" in the array and add that to your output as the "start" value

For the reversal, again we would want to avoid building the entire table for a text as large as the bible. We can do this by using the fact that identical tokens in the first row and last row are always in the same order. This is true because the first row is sorted and the tokens are arranged cyclically: for three consecutive b's in the last row, the tokens after them are sorted, so the b's are sorted. So to reverse:

sort the output tokens. Along with storing the sorted tokens, store the index each token started at. So for the unsorted tokens "nbnaaa", you would store [3 4 5 2 0 1] and "aaabnn". Important: you MUST use a stable sort for this step.

use the "start" value mentioned earlier to rebuild the string:

string decode(string sorted, int[]index, int start){
    string answer = ""+sorted.charAt(start);
    int next = index[start];
    while(next!=start){
        answer = sorted.charAt(next) + answer;
        next = index[next];
    }
    return answer;
}

score 1 · Answer 2 · answered May 14 '11 at 07:19

1

This line:

    String temp = (string.substring(index,string.length()) + string.substring(0,index));

is going to create a copy of the entire input text each time you call it. Since you call it N times for an input text of N characters, your algorithm will be O(N^2).

See if you can optimize the originalSuffix method to avoid that copying.

answered May 14 '11 at 07:19

Stephen C

698,415
94
811
1,216

the copying is necessary in order to create a sorted array of the transforms – nope May 14 '11 at 07:38
No it isn't. Or if it is, your implementation of that method is broken. The method is creating and returning a String that is 6 characters long, but is copying the entire input string in the process. – Stephen C May 14 '11 at 09:32

Optimization of the Burrows Wheeler transform

2 Answers2