0

Background: I'm going to be inserting about a billion key value pairs. I need an in-memory index with which I can simultaneously do look ups for the (32 bit integer) value for a (unique, 64 bit integer) key. There's no updating, no deleting and no traversing. The keys are generally gradually increasing with time.

What index structure is most appropriate to handle this?

The requirements I can think of are:

  • It needs to have efficient rebalancing, due to the increasing keys
  • It needs to use memory efficiently to fit in ram, preferably < 28GB
  • It needs to have very efficient lookups
Max
  • 2,760
  • 1
  • 28
  • 47
  • are the keys are monotonically increasing as are timestamps? – Dan D. Oct 13 '12 at 10:56
  • @DanD. The first 42 bytes of the keys are actually timestamps, but they only come in roughly ordered. So for the keys to come in in a particular minute, most of them will be from the last hour. There will be others however that are from much longer ago. – Max Oct 13 '12 at 11:15
  • I think the best method for this is something like _the log structured merge tree_ – Dan D. Oct 13 '12 at 11:27

1 Answers1

0

There's probably no more efficient datastructure for this problem than a simple sorted vector. (Actually, given alignment issues and depending on access characteristics, you might want to put keys and values in separate vectors.) But there are a number of practical problems, particularly if you don't know how big the data will be. If you do know this, or if you're prepared to just preallocate too much space and then die if you get more data than will fit in this space, then that's fine, although you still need to worry about keeping the vector sorted.

A possibly better approach is to keep a binary search tree of index ranges, where the leaves of the BST point to "clumps" of data (i.e. vectors). (This is essentially a B+ tree.) The clumps can be reasonably large; I'd say something like the amount of data you expect to receive in a couple of minutes, or several thousand entries. They don't have to all be the same size. (B+-trees usually have a smaller fanout than that, but since your data is "mostly sorted", you should be able to use a larger one. Don't make it too large; the only point is to reduce overhead and possibly cache-thrashing.)

Since your data is "mostly sorted", you can accumulate data for a while, keeping it in an ordinary ordered map (assuming you have such a thing), or even in a vector using insertion sort. When this buffer gets large enough, you can append it to your main data structure as a single clump, repartitioning the last clump to deal with overlaps.

If you're reasonably certain that you will only rarely get out-of-order keys, would be to keep a second conventional BST of out-of-order data elements. Any element which cannot be accomodated by repartitioning the new clump and the previous last one can just be added to this BST. To do a lookup, you do a parallel lookup between the main structure and the out-of-order structure.

If you're paranoid or not certain enough about the amount of unordered data, just use the standard B+-tree insertion algorithm, which consists of creating clumps with a little bit of reserved but unused space to allow for insertions (a few per cent; you want to avoid space overhead), and splitting a clump if necessary.

rici
  • 234,347
  • 28
  • 237
  • 341
  • Thanks for your answer. I ended up implementing something a bit similar to a b-tree but designed to cope with increasing numbers. However I then came across Google's sparsehash (http://sparsehash.googlecode.com) and while my tree outperformed it by 50% over 50 million inserts, sparsehash's memory consumption was amazingly low, so I went with that. – Max Oct 14 '12 at 10:39