0

I need to create a WebJob that handles some work that is not super time-sensitive. I use both DocumentDb and Azure Table Storage so I deal with de-normalized data that need to be handled by some backend process to keep consistent.

I have multiple uses for this WebJob so I'm trying to figure out the best way to pass data to it.

Which is the right approach for sending requests to my WebJob?

  1. Create a generic object with properties that can store the data while the request is in the queue. In other words, some type of container that I use for transporting data through the queue.
  2. Persist the data in some backend database and send a simple command via the queue?

One example I can think of is that I have a list of 15 entities that I need to update in my Azure Table Storage. This may take multiple read/write operations which will take time.

If I use approach #1, I'd "package" the list of 15 entities in an object and put it in a message in my queue. Depending on the situation, some messages may get a bit fat which concerns me.

If I use approach #2, I'd save the ID's of the entities in a table somewhere (SQL or Azure Table Storage) and I'd send some type of batch ID via a message. Then my WebJob would receive the batch Id, first retrieve the entities and process them. Though this approach seems like a better one, I'm afraid, this will be pretty slow.

Please keep in mind that the primary use of this particular WebJob is to speed up response times for end users in situations that require multiple backend operations. I'm trying to handle them in a WebJob so that what's time-sensitive gets processed right away and the other "not-so-time-sensitive" operations can be handled by the WebJob.

I want my solution to be both very robust and as fast as possible -- though the job is not highly time sensitive, I still want to process the backend job quickly.

Sam
  • 26,817
  • 58
  • 206
  • 383
  • Most likely this will be closed as off topic for being too broad/opinion based. However I do want to mention one issue with approach 1 - Size of a single message in a queue is limited to 64KB and because it uses 2 bytes to save single character, you're effectively left with 32KB. You may want to think about that with approach 1 as well. – Gaurav Mantri May 17 '16 at 06:47
  • Good point which pretty much eliminates approach #1. That means I need to create a record of what needs to be done - probably in Azure Table Storage - and just send a simple JobId through my queue. My WebJob then reads the JobId first to figure out what needs to be done, then handle the request. Thank you for the very useful point. – Sam May 17 '16 at 07:00
  • 1
    One thing you could is find a middle ground between #1 & #2. Instead of saving the entire data to be processed in a queue, save that in a blob instead and save the blob URL as message contents. A blob in this case can hold up 200GB worth of data. Your webjob will read the contents of the blob and get all the data that needs to be processed in one call instead of making several calls to queue and table storage. – Gaurav Mantri May 17 '16 at 07:15

0 Answers0