-1

I want to use a typical push queue to process slow operations in background. For some HTTP requests, the app creates a new task according to the parameters sent.

My app serves the http request from home automation systems installed at our customers. The number of installed systems will increase by 4000 per month. Each system will communicate every 6 seconds with the server. However, a task will only be created if an event occurs and is embedded in the parameters passed by the query. This occurs approximately 600 times per day. It is important to process the task sent by a system before it sends another one, so before 6 seconds. A task reads and stores data in Datastore by using Objectify, sends emails in case or alert and/or push to mobile phones.

My question is the following :

How to fix the processing rate, bucket-size and max-concurrent-requests values into the queue.xml?

I need to be sure that all requests will be served and all tasks will be processed. But I don't want to pay too much for unused instances.

Thanks in advance for your response.

Victor M Perez
  • 2,185
  • 3
  • 19
  • 22
Robin wood
  • 3
  • 2

2 Answers2

0

For the use case you're mentioning, it might be worthwhile to look at Pub/Sub + CloudFunctions.

If your queue processors are implemented in AppEngine standard and using auto scaling, then they'll scale automatically to handle new load. For some heavy throughput cases, I shard my push queues (e.g. have 36 queues A-Z, 0-9) and then hash based on the object id (convert to base 36 if needed), so that reduces the stress on a given queue.

Saurabh Gupta
  • 86
  • 3
  • Thanks for your response. I looked at the Pub/Sub as you advised me. It seems to be more complicated to push queue but with them I don't care about scaling, right ? – Robin wood Jan 16 '18 at 12:50
0

All answers are in the queue.xml reference. In this example:

<queue-entries>
  <queue>
    <name>fooqueue</name>
    <rate>1/s</rate>
    <retry-parameters>
      .....
    </retry-parameters>
  </queue>
  <queue>
    .....
  </queue>    
</queue-entries>

<queue-entries></queue-entries> is the root element in which you insert the queues and inside the <queue></queue> you can define:

  1. <rate>:

    The value is a number followed by a slash and a unit of time, where the unit is s for seconds, m for minutes, h for hours, or d for days. For example, the value 5/m says tasks will be processed at a rate of 5 times per minute....

  2. <bucket-size>

    Optional ..... [ ]...... If you don't specify bucket_size for a queue, the default value is 5. We recommend that you set this to a larger value because the default size might be too small for many use cases: the recommended size is the processing rate divided by 5 (rate/5).

  3. <max-concurrent-requests>

    Optional. Sets the maximum number of tasks that can be executed simultaneously from the specified queue. The value is an integer. By default, the limit is 1000 tasks per queue.

In your particular case, let's say you're completely sure that you'll only have 600 tasks per day. That is approximately 1 task each 144 seconds. That is your rate. In the document it looks like

<rate>600/d</rate>

See that the other two values are optional-

Victor M Perez
  • 2,185
  • 3
  • 19
  • 22
  • Thanks for your response. There are approximately 600 tasks per day per system. It is the reason of my question : Absorb the increase in the number of system. – Robin wood Jan 16 '18 at 12:55
  • You can write the xml by using java. That way you could change the number dynamically as your traffic increases. look at this: https://www.journaldev.com/1112/how-to-write-xml-file-in-java-dom-parser – Victor M Perez Jan 16 '18 at 13:00