Best Way to Handle Background Processes

Question

I'm trying to implement an iCal synchronization service for my project. There are nearly 10.000+ listings (more to come) in the database and almost every listing has a Google Calendar iCal URL to be synchronized every 12 hours. Synchronizing a single iCal URL takes about 0.5-1sec, so it will take serious time to process 10000+ items.

I'm not sure how to handle the synchronization process. I'm thinking to use Gearman but not sure if it's the best way. If I use Gearman, what would be the PROS and CONS of Gearman? How would I implement Gearman to handle iCal synchronization?

I also found BraincraftedBackgroundProcess written in PHP. I'm not sure if pure PHP can handle such a busy process. I might consider using it too but I'm still trying to figure out what the best way is.

Answer 1

I'm sure you've solved this by now, but Gearman would be possible solution to handle it. The beauty of Gearman is that you write rather simple workers that could have only one function, and that's how to perform the synchronization of one particular iCal URL.

You would then start as many workers as you'd be comfortable with making requests at the same time to Google's Calendar API, so you can easily scale back or up your performance by starting or killing of worker processes.

Updating calendars on regular intervals could be done by having a cron script fetch all calendars that hasn't been updated the last 12 hours, and then adding them to gearmand as tasks to be performed. You can also throttle the amount of requests if you do this every 15 minutes and have a hard limit of fetching only 250 calendars (or something like that) at a time, allowing updates to be bit more fluent that just every 12 hours.

If you need more specific advice on how to implement it using gearmand you'd have to have a more specific question and example code you're using, but there's nothing stopping you from using Gearman to do that task.