0

In my app a user can upload several photographs at once.

Once these are received by the server the following happens:

  • A Gearman worker job is scheduled which processes all the uploaded images, creating various different crop sizes and applying Lancosz filtering before finally writing them to disk. Depending on the number and resolution of photographs uploaded this process may take up to around 10 seconds.
  • Within the main PHP thread the database is updated to reflect the uploaded images.

Although everything works fine, there is no guarantee that the Gearman worker has completed by the time the database has updated. Meaning that a user may potentially get broken image links if they access a page which queries the database for those images.

Therefore I need to do one of the following:

  1. Move my database updating code into the Gearman worker so that it completes once the image processing has finished.
  2. Temporarily, in the main PHP thread, place the full size images in ALL the locations, duplicating the full size image several times into the various differend directories ("/assets/images/cropped/image.jpg", "/assets/images/thumbnail/image.jpg" etc) waiting for them to be overwritten with the smaller cropped and filtered versions later when the Gearman worker completes.

For obvious reasons solution number 1 makes the most sense although of course that will have the drawback of potentially causing a delay between the user uploading the images and he/she actually being able to view them within the application.

Therefore this is really 2-3 questions in one:

  • Is solution 1 or 2 the best way to go, if so why? And is there a much better way of managing this situation that I am not seeing?
  • If I go with solution 1 is there anyway to manage the potential delay caused by updating the database within the worker?
  • If I choose to go with solution 1 I have a problem as I am relatively new to object oriented programming in PHP. As explained below:

If I do indeed go with solution 1 I need to be able to access the public functions for my main Applications database class and memcached class as well as the class of the function which called the Gearman worker itself, all from within the Gearman worker.

For example, in my main app to perform a MySQL PDO database query I use $query = new Query($sql); then bind the values and execute it etc. And at the top of that file I have use \App\Query so that I have access to those public functions. In other instances I would be accessing a function from within another function but in the same class using self::functionToPerform(); The Gearman workers scripts are external to my actual App so how would I go about accessing these functions from within the worker?

I know this is a lot to ask from one question and I hope I have articulated it well enough. Any help really would be appreciated.

gordyr
  • 6,078
  • 14
  • 65
  • 123

1 Answers1

1

How important is real-time upload/update images? Because if time for process images is ~10 sec., user can see notification about this. And as soon as images ready web-page will be updated.

I think workers for images processing must do just this job. You can create workers for DB updates. It's will be two types of workers which can communicate with a message queue (like a RabbitMQ).

And upload process will be looks like this:

  • User uploading his images
  • Image processing worker gets a task and do it
  • Image processing worker sends notify message in queue: 'Images ready!'
  • DB update worker gets this message and do updates
  • DB update worker sends notify message and main process update the page view

Ofcourse, if I right understand a problem...

Kirill Zorin
  • 239
  • 1
  • 5
  • Thanks, this is essentially the same as my solution number 1, but this still leaves me not knowing how to access my apps classes and public functions from within a gearman worker. Thanks though :-) – gordyr Feb 21 '13 at 06:43
  • Hmm, you can copied class to the workers server, I think you know) And in other way you can use RPC-like protocol for communication between workers and general system. – Kirill Zorin Feb 21 '13 at 11:33