6

I need to perform a well-known Burrows-Wheeler Transform in linear time. I found a solution with suffix sorting and EOF character, but appending EOF changes the transformation. For example: consider the string bcababa and two rotations

  • s1 = abababc
  • s2 = ababcab

it's clear that s1 < s2. Now with an EOF character:

  • s1 = ababa#bc
  • s2 = aba#bcab

and now s2 < s1. And the resulting transformation will be different. How can I perform BWT without EOF?

Gilles 'SO- stop being evil'
  • 104,111
  • 38
  • 209
  • 254
user8078
  • 161
  • 3

3 Answers3

5

You can perform the transform in linear time and space without the EOF character by computing the suffix array of the string concatenated with itself. Then iterate over the suffix array. If the current suffix array value is less than n, add to your output array the last character of the rotation starting at the position denoted by the current value in the suffix array. This approach will produce a slightly different BWT transform result, however, since the string rotations aren't sorted as if the EOF character were present.

A more thorough description can be found here: http://www.quora.com/Algorithms/How-I-can-optimize-burrows-wheeler-transform-and-inverse-transform-to-work-in-O-n-time-O-n-space

John Kurlak
  • 6,594
  • 7
  • 43
  • 59
  • 1
    Is using an EOF character equivalent to the string concatenated with itself method? The output I get don't seem to be identical. Using an EOF character, "\0", which should be lower than all other characters, I get `ptr: 13, string: "CTAAAACACGAGA\0GATGCAGGTATTTTATGTTAGTGATGCATTTTATGGCTCCCCGAGCATATC"` Using the concatenated input method, I get, `ptr: 12, str: "TAAAACACGAGACGATGCGGATATTTTATGTTAGTGATGCATTTTATGGCTCCCCGAGCATATC"` Even if we ignore the NUL terminator, the output is still different. – DSnet Nov 24 '15 at 12:12
  • @DSnet What input string did you use? AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTCTCTGAC? – John Kurlak Nov 24 '15 at 16:39
  • @DSnet... OK, I'm able to reproduce your results. I'll investigate later if I have time. Thanks! – John Kurlak Nov 24 '15 at 17:30
  • That is the correct string, thanks for doing this. I'm trying to write a bzip2 encoder using SAIS. Currently, with the duplicated input string trick, I seem to be able to generate valid bzip2 files. However, the slowdown due to needing to process 2n the actual input is unfortunate. – DSnet Nov 25 '15 at 11:15
  • @DSnet Yep... my approach is definitely wrong. The problem is that the string rotations need to be sorted in a way where the EOF character is lexicographically smaller than the other characters. My approach without the EOF character doesn't take this into account when it sorts the suffixes, so the sort order is wrong. This causes it to produce the wrong output. I've since updated my Quora answer. I'm wondering--if you're just writing a bzip2 encoder, why do you need to produce the BWT without an EOF character? Why not just do the first approach I suggest? – John Kurlak Nov 25 '15 at 16:18
  • @DSnet You might also find this interesting: http://rd.springer.com/chapter/10.1007%2F978-3-642-03784-9_9 – John Kurlak Nov 25 '15 at 16:23
1

You need to have EOF character in the string for BWT to work, because otherwise you can't perform the inverse transform to get the original string back. Without EOF, both strings "ba" and "ab" have the same transformed version ("ba"). With EOF, the transforms are different

ab        ba

a b |     a | b
b | a     b a |
| a b     | b a

i.e. ab transforms to "|ab" and ba to "b|a".

EOF is needed for BWT because it marks the point where the character cycle starts.

Re: doing it without the EOF character, according to Wikipedia,

Since any rotation of the input string will lead to the same transformed string, the BWT cannot be inverted without adding an 'EOF' marker to the input or, augmenting the output with information, such as an index, that makes it possible to identify the input string from the class of all of its rotations.

There is a bijective version of the transform, by which the transformed string uniquely identifies the original. In this version, every string has a unique inverse of the same length.

The bijective transform is computed by first factoring the input into a non-increasing sequence of Lyndon words; such a factorization exists by the Chen–Fox–Lyndon theorem, and can be found in linear time. Then, the algorithm sorts together all the rotations of all of these words; as in the usual Burrows–Wheeler transform, this produces a sorted sequence of n strings. The transformed string is then obtained by picking the final character of each of these strings in this sorted list.

Antti Huima
  • 25,136
  • 3
  • 52
  • 71
  • I would solve the problem with EOF character. If I would have a student who would solve the problem without EOF character because "he/she could only find a solution without it", I would fail that student. – Antti Huima May 10 '12 at 05:33
  • the check is performed by automatic system, if I use EOF I'll have a "wrong answer". – user8078 May 10 '12 at 05:54
  • Hmm interesting :) then you just need to perform it without EOF. I'm suspicious of the linear time requirement, though, if this is a homework / exercise / interview question, because even simple sorting is not a linear-time operation. – Antti Huima May 10 '12 at 06:02
  • @antti.huima You can do it in linear time and space if you work with the suffix array of the string concatenated with itself. See my answer for more details. – John Kurlak Dec 21 '12 at 01:23
  • 3
    You are not correct. Practical implementations of BWT do not include the sentinel explicitly in the output. Instead, the index of the original string is passed along with the output for decoding (unless you use the bijective version, which is slower and far less known). Thus, you can do inverse transform of the input encoded without a sentinel, you just need that original string index. – kvark Feb 26 '14 at 14:31
0

I know this thread is quite old but I had the same problem and came up with the following solution:

  • Find the lexicographical minimal string rotation and save the offset (needed to reverse) (I use the lydon factorization)
  • Use the normal bwt algorithms on the rotated string (this produces the right output because all algos asume that the string is followed by the lexicographically minimal char)
  • To reverse: unbwt using e.g. backward search starting at index 0 and write the corrosponding char to the saved offset
Christoph Diegelmann
  • 2,004
  • 15
  • 26