Storing huge randomized list of similar items in Redis

Question

Redis 2.0.3

I need to store a huge list of items in Redis. Each item is a short string (less than 256 characters).

I need to do two operations on the list:

Add many (several thousands to a million) identical items. (Several times a day)
Remove one random item from the list. It is not necessary to have "fair" random. Any "good enough" approach will do. (Up to several hundred times a second)

I do not have enough RAM to store all items in a list one by one.

I think that I need to store items in batches, name and a counter. (There will be up to several thousands distinct items, more like several hundreds.)

But I'm not sure how to organize this effectively.

Any hints?

score 0 · Accepted Answer · answered Feb 25 '11 at 14:52

Well, since nobody is up to help me, here is a "dumb" solution, in pseudocode.

Get random element:

function maybe_get_next_item()
  item_name = SRANDMEMBER "items-set"
  item_key = "items:" + item_name

  new_item_count = DECR (item_key)

  if new_item_count < 0 then
    LOCK -- As explained in SETNX docs
      new_item_count = GET (item_key) -- More added while we were locking?
      if new_item_count and new_item_count < 0 then
        SREM (item_name) -- No, expire it
      end
    UNLOCK
  end

  if new_item_count and new_item_count >= 0 then
    return item_name
  end

  return false -- this item not found
end

function get_next_item()
  item_name = maybe_get_next_item()
  while not item_name and (SCARD "items-set" > 0) do
    item_name = maybe_get_next_item()
  end
  return item_name -- false if all items are expended
end

Insert new elements

function insert_items(item_name, amount)
  LOCK -- As explained in SETNX docs
    SADD "items-set" (item_name)
    INCRBY ("items:" + item_name) amount
  UNLOCK
end

Please do suggest a better solution if it exists, I'm still groking Redis, and may miss something obvious.

I suspect that LOCK/UNLOCK in insert_items() may be superfluous and can be replaced with MULTI/EXEC, but I think that it is needed for LOCK/UNLOCK in maybe_get_next_item() to work properly (which I do not know how to replace with MULTI/EXEC)...

Note that probabilities here are skewed, as number of identical items does not play any part in the chance to get the item. This is okay for my particular use-case. — Alexander Gladysh, Feb 25 '11 at 18:06
On 2.2 better approach would be to use WATCH: http://stackoverflow.com/questions/5118807/redis-how-to-atomically-remove-an-item-from-set-if-a-counter-in-another-key-is-b — Alexander Gladysh, Feb 28 '11 at 04:35

Storing huge randomized list of similar items in Redis

1 Answers1