Java + Membase + spymemcached bulk operations

Question

I am currently writing an application that requires bulk operations on a key/value store, at this time I am using membase.

spymemcached allows bulk get, but not bulk CAS or add; features that I think would be widely used if implemented.

At this point my code for a set of bulk operations is roughly as shown below.

"client" is a single MemcachedClient.

ArrayList<Future<Boolean>> futures = new ArrayList<Future<Boolean>>();
ArrayList<String> bulkGet = new ArrayList<String>();
for(int i=0; i<50; i++){
    String key = "Key_" + x + "_" + i;
    Future<Boolean> fut = client.add(key, 0, "Value_" + x + "_" + i);
    futures.add(fut);
    bulkGet.add(key);
}

int count = 0;
for(Future<Boolean> fut : futures){
    if(fut.get()==true){
        count++;
    }
}
System.out.println("Added " + count + " records.");

Map<String,Object> bulkGot = client.getBulk(bulkGet);
System.out.println("Retrieved " + bulkGot.size() + " records");

The blocking call on Future.get() seems highly inefficient, is there a better way? In my actual scenario I'd like the ability to handle the futures as soon as they return (which may or may not be in the order in which they were sent?).

Also, are the following operations possible (or planning to be implemented)?

-Add or return existing value

-Delete if value equals a known value

-Set if value equals a known value

Thanks, Marcus

score 0 · Accepted Answer · answered Aug 18 '11 at 00:16

I'll try to address as many things here as possible.

Spymemcached allows bulk get, but not bulk CAS or add; features that I think would be widely used if implemented.

I completely agree here, but we probably wouldn't go about this by adding bulkAdd, bulkCas, ... functions to MemcachedClient. Spymemcached currently, by default optimizes get and set operations by sending them as bulk sets or bulk gets whenever it can. There are plans to extend this to all memcached operations, but it hasn't been done yet. When it is though it will happen behind the scenes and probably won't be directly available to spymemcached users.

The blocking call on Future.get() seems highly inefficient?

Yes, which I will address below, but also note that if this was a bulk operation then your code would still block for a little bit. With individual add operations you code would probably only block for the first operation because after that all of the other operations will have already finished so calls to Future.get() after the first call would return immediately.

So is there a better way?

You can extend the MemcachedClient class and copy one of the existing functions and add your own code to the callback function.

Also, are the following operations possible (or planning to be implemented)?

Spymemcached implements all of the functionality that memcached offers. The operations you mention can already be build with the the functionality Spymemcached has.

-Add or return existing value

add keyA. If it fails get keyA.

-Delete if value equals a known value

get keyA. If keyA equals a known value, delete keyA

-Set if value equals a known value

get keyA. If keyA equals a known value, set keyB.

If you are looking to get new options added to memcached then you should post your requests on memcached.org. To be honest though, the memcached guys like to keep the api as small as possible so I don't think it is likely these will get addded.

Java + Membase + spymemcached bulk operations

1 Answers1