App Engine: Memcache design to ensure fast read and high write throughput

Question

I'm currently using App Engine with Python.

My application looks like a massive multiplayer game. I would like to improve the time necessary to fetch the latest actions published in a "room".

I already use the Memcache API to store and retrieve actions that have a low write throughput (1 every minute). Besides, I'm also considering it for retrieving actions that have a high write throughput (several per seconds if there are a lot of people in the "room": e.g. messages published by players)

How would you advise me to design the memcache storage for this high write throughput? A single key/value pair where value = a list of the latest published actions does not seem like the right solution.

Thanks,

Answer 1

It is hard to answer the question without more details of your application. A simple idea is to use more key/value pair.

For example:

# when write action log
# 1. decide the prefix based on the data
prefix = decide_prefix(action)

# 2. random pick up a bulk for this data
index = random.choice(range(100))

# 3. write action into a queue
action_list = memceche.get("%s_%s"%(prefix, index))
action_list.append(action)
memcache.set("%s_%s"%(prefix,index), action_list)

# when read action log
action_lists = memcache.get_multi(["%s_%s"%(prefix, k) for k in range(100)])

# merge all action list together.
all_action_list = merge_action_list(action_lists)

Moreover, you can use compare and set to handle the concurrent request. http://code.google.com/appengine/docs/python/memcache/overview.html#Using_Compare_and_Set_in_Python

GAE memcahce has its own limit and doesn't guarantee data persistence. Increase memcache usage may produce more data lost. Thus, you still need to use datastore to save persistence data.

http://code.google.com/appengine/docs/python/memcache/overview.html#Quotas_and_Limits