Stack Overflow
Stack Overflow

Questions tagged [task-queue]

With the Task Queue API, applications can perform work outside of a user request, initiated by a user request.

Tasks queues are an efficient and powerful tool for background processing; they allow your application to define tasks, add them to a queue, and then use the queue to process them in aggregate. You name queues and configure their properties in a configuration file named queue.yaml. A queue can be one of two types—push or pull—based on your needs.

Push queues function only within the App Engine environment. These queues are the best choice for applications whose tasks work only with App Engine tools and services. With push queues, you simply configure a queue and add tasks to it. App Engine handles the rest. Push queues are easier to implement, but are restricted to use within App Engine. For more information about push queues and examples of how to use them, see Using Push Queues.

In summary, push queues allow you to process tasks within App Engine at a steady rate and App Engine scales computing resources according to the number of tasks in your queue.

757 questions
15
votes
2 answers

How to implement a light pub-sub service on App Engine?

During his Google I/O 2009 "Offline processing on App Engine: A look ahead" presentation (video, slides), Brett Slatkin presents the Task Queue service. He states that Pub-sub systems maximize transactions, decoupling: Large numbers of small…
nulltoken
  • 64,429
  • 20
  • 138
  • 130
15
votes
3 answers

How to create multiple workers in Python-RQ?

We have recently forced to replace celery with RQ as it is simpler and celery was giving us too many problems. Now, we are not able to find a way to create multiple queues dynamically because we need to get multiple jobs done concurrently. So…
RazorHead
  • 1,460
  • 3
  • 14
  • 20
14
votes
1 answer

with node.js on app engine, is it better to use task queues or pub/sub

We have been moving our apis from python to node. We have used task queues with our Python app engine apis. With node.js now supported on app engine, do you suggest we use task queues or cloud pub/sub for tasks? What would the pros / cons be for…
philipfc
  • 609
  • 9
  • 20
13
votes
3 answers

How can I queue a task to Celery from C#?

As I understand message brokers like RabbitMQ facilitates different applications written in different language/platform to communicate with each other. So since celery can use RabbitMQ as message broker, I believe we can queue task from any…
Joe
  • 155
  • 1
  • 4
13
votes
1 answer

Unsatisfactory job push performance with Python RQ

Trying to use python-rq to support the back end to our Web application, but pushing new jobs takes very long - up to 12 seconds. The performance hit happens when executing the enqueue_call function call, particularly when the number of worker…
Juan Carlos Coto
  • 11,900
  • 22
  • 62
  • 102
12
votes
1 answer

Flask with Celery - Application context not available

I have a Flask application, which is registered as follows: APP = Flask(__name__) APP.config.from_object('config') I have defined a view for a URL, in which a function is called, which interacts with the DB. from tasks import…
Rutwick Gangurde
  • 4,772
  • 11
  • 53
  • 87
11
votes
3 answers

Google App Engine: task_retry_limit doesn't work?

I have a Python GAE app. I want my tasks to stop running or just retry once if they fail. Right now, they run forever despite what my yaml file is telling them! Here is a queue.yaml entry: - name: globalPurchase rate: 10/s bucket_size: 100 …
Barbara
  • 111
  • 1
  • 3
11
votes
6 answers

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build XXXXXXXXXXXX status: FAILURE

I had this error for almost an hour but found a solution resulting into a big worrying dilemma. The solution let to realization that one cannot run gcloud app deploy while there is a task in the task queue to be served by the server version to be…
Geek Guy
  • 710
  • 1
  • 8
  • 28
11
votes
3 answers

How to run a Laravel Job at specify Queue

I have a Job to perform send SMS to user. I want to run this job on the specify queue-name. For example, this job added to "SMS" queue. So I found a way to do this but it's exists some errors. Create job instance and use onQueue() function to do…
Mesuti
  • 878
  • 1
  • 13
  • 29
11
votes
1 answer

Is there a compelling reason to use an AMQP based server over something like beanstalkd or redis?

I'm writing a piece to a project that's responsible for processing tasks outside of the main application facing data server, which is written in javascript using Node.js. It needs to handle tasks which are scheduled in the future and potentially…
Nathan C. Tresch
  • 938
  • 10
  • 24
9
votes
3 answers

Checking status of Task Queue in Google App Engine

I'm putting several tasks into a task queue and would like to know when the specific tasks are done. I haven't found anything in the API about call backs, or checking the status of a task, so I thought I'd see what other people do, or if there's a…
Parker
  • 8,539
  • 10
  • 69
  • 98
9
votes
1 answer

Web Application Architecture - Job/Task Queue needed?

I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time resolution that can be used for scheduling will be one…
Simon
  • 1,643
  • 2
  • 17
  • 23
9
votes
3 answers

asynchronous processing with PHP - one worker per job

Consider a PHP web application whose purpose is to accept user requests to start generic asynchronous jobs, and then create a worker process/thread to run the job. The jobs are not particularly CPU or memory intensive, but are expected to block on…
Josh Johnson
  • 8,832
  • 4
  • 25
  • 31
9
votes
3 answers

Laravel 5: Multiple workers - Cannot delete job XXXXXX: NOT_FOUND

I'm creating a website using Laravel 5. I'm using queues with beanstalkd with 2 tubes. I have 10 workers monitored by daemontools. 5 workers process tube A and B. The other 5 workers only process tube B. Things work great most of the time, but…
Kebian
  • 664
  • 6
  • 17
9
votes
1 answer

Managing Celery Task Results

I am pretty new to Celery and I thought I had read somewhere that the task results only stay around for a limited time. However my backend (redis) is getting pretty bloated after running a lot of tasks through it. Is there a way to set a TTL on…
jwarzech
  • 6,596
  • 11
  • 52
  • 72
1
2
3
…
50 51