4

I am looking for a method to do fast nearest neighbour (hopefully O(log n)) for high dimensional points (typically ~11-13 dimensional). I would like it to behave optimally during insertions after having initialized the structure. KD tree came to my mind but if you do not do bulk loading but do dynamic insertions, then kd tree ceases to be balanced and afaik balancing is an expensive operation.

So, I wanted to know what data structures would you prefer for such kind of setting. You have high dimensional points and you would like to do insertions and query for nearest neighbour.

I J
  • 805
  • 3
  • 9
  • 19

4 Answers4

5

The Curse of Dimensionality gets in the way here. You might consider applying Principal Component Analysis (PCA) to reduce the dimensionality, but as far as I know, nobody has a great answer for this.

I have dealt with this type of problem before (in audio and video fingerprinting), sometimes with up to 30 dimensions. Analysis usually revealed that some of the dimensions did not contain relevant information for searches (actually fuzzy searches, my main goal), so I omitted them from the index structures used to access the data, but included them in the logic to determine matches from a list of candidates found during the search. This effectively reduced the dimensionality to a tractable level.

I simplified things further by quantizing the remaining dimensions severely, such that the entire multidimensional space was mapped into a 32-bit integer. I used this as the key in an STL map (a red-black tree), though I could have used a hash table. I was able to add millions of records dynamically to such a structure (RAM-based, of course) in about a minute or two, and searches took about a millisecond on average, though the data was by no means evenly distributed. Searches required careful enumeration of values in the dimensions that were mapped into the 32-bit key, but were reliable enough to use in a commercial product. I believe it is used to this day in iTunes Match, if my sources are correct. :)

The bottom line is that I recommend you take a look at your data and do something custom that exploits features in it to make for fast indexing and searching. Find the dimensions that vary the most and are the most independent of each other. Quantize those and use them as the key in an index. Each bucket in the index contains all items that share that key (there will likely be more than one). To find nearest neighbors, look at "nearby" keys and within each bucket, look for nearby values. Good luck.

p.s. I wrote a paper on my technique, available here. Sorry about the paywall. Perhaps you can find a free copy elsewhere. Let me know if you have any questions about it.

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Randall Cook
  • 6,728
  • 6
  • 33
  • 68
  • Thanks, but I would not like to reduce the dimensionality of the data since I want exact kNN in the original space. – I J Apr 03 '12 at 01:37
  • Fair enough, though I never saw the value in searching for a fixed number of neighbors at varying distance. Searching for a variable number of neighbors at a fixed distance always seemed more practical to me. – Randall Cook Apr 03 '12 at 06:18
  • @Randall, nice. Does "32-bit key in an STL map" mean exact match, or also 32 1-bit nighhbors ? Any ideas on [bit-string-nearest-neighbour-searching](http://stackoverflow.com/questions/9959728/bit-string-nearest-neighbour-searching) -- looks NP-complete ? – denis Apr 15 '12 at 10:31
  • @Denis, yes, the 32-bit key in an STL map means exact matches, which implies that the fuzzy searches must try several 32-bit values. I read your linked answer to the bit stream search question. Intriguing. I think it's NP-complete. Good stuff. – Randall Cook Apr 15 '12 at 17:17
5

Another data structure that comes to mind is the cover tree. Unlike KD trees which were originally developed to answer range queries, this data structure is optimal for nearest neighbor queries. It has been used in n-body problems that involve computing the k nearest neighbors of all the data points. Such problems also occur in density estimation schemes (Parzen windows). I don't know enough about your specific problem, but I do know that there are online versions of this data structure. Check out Alexander Gray's page and this link

killogre
  • 1,730
  • 15
  • 26
3

If you use a Bucket Kd-Tree with a reasonably large bucket size it lets the tree get better idea of where to split when the leaves get too full. The guys in Robocode do this under extremely harsh time-constraints, with random insertions happening on the fly and kNN with k>80, d > 10 and n > 30k in under 1ms. Check out this kD-Tree Tutorial which explains a bunch of kD-Tree enhancements and how to implement them.

jkflying
  • 1,079
  • 7
  • 15
1

In my experience, 11-13 dimensions is not too bad -- if you bulk-load. Both bulk-loaded R-trees (in contrast to k-d-trees these remain balanced!) and k-d-trees should still work much better than linear scanning.

Once you go fully dynamic, my experiences are much worse. Roughly: with bulk loaded trees I'm seeing 20x speedups, with incrementally built R-trees just 7x. So it does pay off to frequently rebuild the tree. And depending on how you organize your data, it may be much faster than you think. The bulk load for the k-d-tree that I'm using is O(n log n), and I read that there is a O(n log log n) variant, too. With a low constant factor. For the R-tree, Sort-Tile-Recursive is the best bulk load I have seen so far, and also O(n log n) with a low constant factor.

So yes, in high-dimensionality I would consider to just reload the tree from time to time.

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194