3

We push out alerts from GAE, and let's say we need to push out 50 000 alerts to CD2M (Cloud 2 Device Messaging). For this we:

  1. Read all who wants alerts from the datastore
  2. Loop through and create a "push task" for each notification

The problem is that the creation of the task takes some time so this doesn't scale when the user base grows. In my experience we are getting 20-30 seconds just creating the tasks when there is a lot of them. The reason for one task pr. push message is so that we can retry the task if something fails and it will only affect a single subscriber. Also C2DM only supports sending to one user at a time.

Will it be faster if we:

  1. Read all who wants alerts from the datastore
  2. Loop through and create a "pool task" for each 100 subscribers
  3. Each "Pool task" will generate 100 "push tasks" when they execute

The task execution is very fast so in our scenario it seems like the creation of the tasks is the bottleneck and not the execution of the tasks. That's why I thought about this scenario to be able to increase the parallelism of the application. I would guess this would lead to faster execution but then again I may be all wrong :-)

Christer Nordvik
  • 2,518
  • 3
  • 35
  • 52

2 Answers2

1

We do something similar with APNS (Apple Push Notification Server): we create a task for a batch of notifications at a time (= pool task as you call it). When task executes, we iterate over a batch and send it to push server.

The difference with your setup is that we have a separate server for communicating with push, as APNS only supports socket communication.

The only downside is if there is an error, then whole task will be repeated and some users might get two notifications.

Peter Knego
  • 79,991
  • 11
  • 123
  • 154
  • We use Urban Airship for the iOS notifications and they are really fast, but for Android we are doing it ourselves and that's where our delay is. But since you have implemented what I am thinking about doing then I hope we are both thinking the right thing, so I'll accept that answer :-) – Christer Nordvik Mar 11 '12 at 21:32
1

This sounds like it varies based on the number of alerts you need to send out, how long it takes to send each alert, and the number of active instances you have running.

My guess is that it takes a few milliseconds to tens of milliseconds to send out a CD2M alert, while it takes a few seconds for an instance to spin up, so you can probably issue a few hundred or a few thousand alerts before justifying another task instance. The ratio of the amount of time it takes to send each CD2M message vs the time it takes to launch an instance will dictate how many messages you'd want to send per task.

If you already have a fair number of instances running though, you don't have the delay of waiting for instances to spin up.

BTW, this seems almost like a perfect application of the MapReduce API. It mostly does what you describe in the second version, except it takes your initial query, and breaks that up into subqueries that each return a "page" of the result set. A task is launched for each subquery which processes all the items in its "page". This is an improvement from what you describe, because you don't need to spend the time looping through your initial result set.

I believe the default implementation for the MapReduce API just queries for all entities of a particular kind (ie all User objects), but you can change the filter used.

dragonx
  • 14,963
  • 27
  • 44