Fastest way to compute all 1-hamming distanced neighbors of strings?

Question

I am trying to compute hamming distances between each node in a graph of n nodes. Each node in this graph has a label of the same length (k) and the alphabet used for labels is {0, 1, *}. The '*' operates as a don't care symbol. For example, hamming distances between labels 101*01 and 1001*1 is equal to 1 (we say they only differ at the 3rd index).

What I need to do is to find all 1-hamming-distance neighbors of each node and report exactly at which index those two labels differ.

I am comparing each nodes label with all others character by character as follows:

    // Given two strings s1, s2
    // returns the index of the change if hd(s1,s2)=1, -1 otherwise.

    int count = 0;
    char c1, c2;
    int index = -1;

    for (int i = 0; i < k; i++)
    {
        // do not compute anything for *
        c1 = s1.charAt(i);
        if (c1 == '*')
            continue;

        c2 = s2.charAt(i);
        if (c2 == '*')
            continue;

        if (c1 != c2)
        {
            index = i;
            count++;

            // if hamming distance is greater than 1, immediately stop
            if (count > 1)
            {
                index = -1;
                break;
            }
        }
    }
    return index;

I may have a couple of millions nodes. k is usually around 50. I am using JAVA, this comparison takes n*n*k time and operates slow. I considered making use of tries and VP-trees but could not figure out which data structure works for this case. I also studied the Simmetrics library but nothing flashed into my mind. I would really appreciate any suggestions.

Answer 1

Try this approach:

Convert the keys into ternary numbers (base 3). i.e. 0=0, 1=1, *=2 10 digits ternary give you a range of 0..59049 which fits in 16 bits.

That means two of those would form a 32 bit word. Create a lookup table with 4 billion entries that return the distance between those two 10 digit ternary words.

You can now use the lookup table to check 10 characters of the key with one lookup. If you use 5 characters, then 3^5 gives you 243 values which would fit into one byte, so the lookup table would only be 64 KB.

By using shift operations, you can create lookup tables of different sizes to balance memory and speed.

That way, you can optimize the loop to abort much more quickly.

To get the position of the first difference, you can use a second lookup table which contains the index of the first difference for two key substrings.

If you have millions of nodes, then you will have many that start with the same substring. Try to sort them into buckets where one bucket contains nodes that start with the same key. The goal here is to make the buckets as small as possible (to reduce the n*n).

Answer 2

Instead of / additional to the string, store a mask for 1 bits and a mask for * bits. One could use BitSet, but let's try without.

static int mask(String value, char digit) {
    int mask = 0;
    int bit = 2; // Start with bits[1] as per specification.
    for (int i = 0; i < value.length(); ++i) {
        if (value.charAt(i) == digit) {
            mask |= bit;
        }
        bit <<= 1;
    }
    return mask;
}

class Cell {
    int ones;
    int stars;
}

int difference(Cell x, Cell y) {
    int distance = 0;
    return (x.ones & ~y.stars) ^ (y.ones & ~x.stars);
}

int hammingDistance(Cell x, Cell y) {
    return Integer.bitCount(difference(x, y));
}

boolean differsBy1(Cell x, Cell y) {
    int diff = difference(x, y);
    return diff == 0 ? false : (diff & (diff - 1)) == 0;
}

int bitPosition(int diff) {
    return Integer.numberOfTrailingZeroes(diff);
}

Answer 3

Interesting problem. It would be easy it weren't for the wild card symbol.

If the wildcard was a regular character in the alphabet, then for a given string you could enumerate all k hamming distance 1 strings. Then look these strings up in a multi-map. So for example for 101 you look up 001,111 and 100.

The don't care symbol makes it so that you can't do that lookup. However if the multi-map is build such that each node is stored by all its possible keys you can do that lookup again. So for example 1*1 is stored as 111 and 101. So when you do the look up for 10* you look up 000,010,011,001,111 which would find 1*1 which was stored by 111.

The upside of this is also that you can store all labels as integers rather then trinary structures so with an int[3] as the key value you can use any k < 96.

Performance would depend on the backing implementation of the multi-map. Ideally you'd use a hash implementation for key sizes < 32 and a tree-implementation for anything above. With the tree-implementation all nodes be connected to their distance-1 neighbors in O(n*k*log(n)). Building the multi-map takes O(n * 2 ^ z) where z is maximum number of wildcard characters for any string. If the average number of wildcards is low this should be an acceptable performance penalty.

edit: You improve look up performance for all nodes to O(n*log(n)) by also inserting the hamming distance 1 neighbors into the multi-map but that might just explode its size.

Note: I'm typing this in a lunch break. I haven't checked the details yet.