Collapsing set of strings based on a given hamming distance

Question

Given a set of strings (first column) along with counts (second column), e.g.:

aaaa 10
aaab 5
abbb 3
cbbb 2
dbbb 1
cccc 8

Are there any algorithms or even implementations (ideally as a Unix executive, R or python) which collapse this set into a new set based on a given hamming distance.

• Collapsing implies adding the count
• Strings with a lower count are collapsed into strings with higher counts.

For example say for hamming distance 1, the above set would collapse the second string aaab into aaaa since they are 1 hamming distance apart and aaaa has a higher count. The collapsed entry would have the combined count, here aaaa 15

For this set, we'd, therefore, get the following collapsed set:

aaaa 15
abbb 6
cccc 8

Ideally, an implementation should be efficient, so even heuristics which do not guarantee an optimal solution would be appreciated.

Further background and motivation

Calculating the hamming distance between 2 strings (a pair) is been implemented in most programming languages. A brute force solution would compute compute the distance between all pairs. Maybe there is no way around it. However e.g. I'd imagine an efficient solutions would avoid calculating the distance for all pairs etc. There are maybe clever ways to save some calculations based on metric theory (since hamming distance is a metric), e.g. if hamming distance between x and z is 3, and x and y is 3, I can avoid calculating between y and z. Maybe there is a clever k-mer approach, or maybe some efficient solution for a constant distance (say d=1).

Even if it there was only a brute force solution, I'd be curious if this has been implemented before and how to use it (ideally without me having to implement it myself).

Answer 1

I thought up the following:

This reports the item with the highest score with the sum of its score and the scores of its near by neighbors. Once a neighbor is used it is not reported separately.

I suggest using a Vantage-point tree as the metric index.

The algorithm would look like this:

1. construct the metric index from the strings and their scores
2. construct the max heap from the strings and their scores
3. for the string with the highest score in the max heap:
4. use the metric index to find the near by strings
5. print the string, and the sum of its score and its near by strings
6. remove from the metric index the string and each of the near by strings
7. remove from the max heap the string and each of the near by strings
8. repeat 3-7 until the max heap is empty

Perhaps this could be simplified by using a used table rather than removing anything. The metric space index would not need to have efficient deletion nor would the max heap need to support deletion by value. But this would be slower if the neighborhoods are large and overlap frequently. So efficient deletion might be a necessary difficulty.

1. construct the metric index from the strings and their scores
2. construct the max heap from the strings and their scores
3. construct the used table from an empty set
4. for the string with the highest score in the max heap:
5. if this string is in the used table: start over with the next string
6. use the metric index to find the near by strings
7. remove any near by strings that are in the used table
8. print the string, and the sum of its score and its near by strings
9. add the near by strings to the used table
10. repeat 4-9 until the max heap is empty

I can not provide a complexity analysis.

I was thinking about the second algorithm. The part that I thought was slow was the checking of the neighborhood against used table. This is not needed as deletion from a Vantage-point tree can be done in linear time. When searching for the neighbors, remember where they were found and then remove them later using these locations. If a neighbor is used as a vantage-point, mark it as removed so that a search will not return it, but leave it alone otherwise. This I think restores it to below quadratic. As otherwise it would be something like number of items times size of neighborhood.

---

In response to the comment. The problem was "Strings with a lower count are collapsed into strings with higher counts." as such this does compute that. It is not a greedy approximation that could result non-optimal result as there was nothing to maximize or minimize. It is an exact algorithm. It returns the the item with the highest score combined with the score of its neighborhood.

This can be viewed as assigning a leader to each neighborhood such that each item has at most one leader and that leader has the largest overall score so far. This can be viewed as a directed graph.

The specification wasn't for dynamic programming or optimization problem. For that you would ask for the item with the highest score in the highest total scoring neighborhood. That can also be solved in a similar way by changing the ranking function strings from its score to the pair of the sum of its score and its neighborhood, and its score.

It does mean that it can't be solved with a max heap over the scores as removing items affects the neighbors of the neighborhood and one would have to recalculate their neighborhood score before again finding the item with the highest total scoring neighborhood.