Stack Overflow
Stack Overflow

Questions tagged [task-queue]

With the Task Queue API, applications can perform work outside of a user request, initiated by a user request.

Tasks queues are an efficient and powerful tool for background processing; they allow your application to define tasks, add them to a queue, and then use the queue to process them in aggregate. You name queues and configure their properties in a configuration file named queue.yaml. A queue can be one of two types—push or pull—based on your needs.

Push queues function only within the App Engine environment. These queues are the best choice for applications whose tasks work only with App Engine tools and services. With push queues, you simply configure a queue and add tasks to it. App Engine handles the rest. Push queues are easier to implement, but are restricted to use within App Engine. For more information about push queues and examples of how to use them, see Using Push Queues.

In summary, push queues allow you to process tasks within App Engine at a steady rate and App Engine scales computing resources according to the number of tasks in your queue.

757 questions
9
votes
1 answer

How to limit throughput with RabbitMQ?

Where did the question: We are using RabbitMQ as task queue. One of specific tasks - sending notices to Vkontakte social network. They api has limit to request per seconds and this limit based on your application size. Just 3 calls for app with less…
Nick Bondarenko
  • 6,211
  • 4
  • 35
  • 56
9
votes
2 answers

A queueing system for Perl

I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs. I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so…
Eugene Yarmash
  • 142,882
  • 41
  • 325
  • 378
8
votes
3 answers

Creating a execution queue by using Task.ContinueWith?

I have several actions that I want to execute in the background, but they have to be executed synchronously one after the other. I was wondering if it's a good idea to use the Task.ContinueWith method to achieve this. Do you foresee any problems…
lukebuehler
  • 4,061
  • 2
  • 24
  • 28
8
votes
2 answers

accessing bull-queue to view job stats from nodejs

I need to access bull-queue to view job stats and show on the page. I'm using bull-repl to access the queue from CLI, as follows: > bull-repl BULL-REPL> connect marathon reddis://localhost:6379 Connected to reddis://localhost:6379, queue:…
Yury Stanev
  • 419
  • 1
  • 5
  • 16
8
votes
2 answers

Python: Distributed task queue for different specific workers

I am looking for a python library / framework that manages task distribution (e.g. a task queue). However, tasks will require specialized workers: Worker A can only handle tasks of type a, workers B and C only of type b etc. Also, these workers will…
mrspl
  • 481
  • 1
  • 5
  • 10
8
votes
2 answers

Google App Engine: bucket_size parameter on Task Queues

Google documentation on Bucket_size parameter gives just a brief definition, a Wikipedia link and an example: bucket_size Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution…
systempuntoout
  • 71,966
  • 47
  • 171
  • 241
8
votes
2 answers

What's the execution model of uwsgi spoolers?

I need a task queue so that some of the heavy operations can be moved out of the uwsgi context without affecting the users. Since we're already using uwsgi app server, it'd be desirable if uwsgi spooler could be used as a task queue. I wanted to…
dotgc
  • 500
  • 1
  • 7
  • 21
8
votes
1 answer

How to correctly catch and process RQ timeouts in Python?

Trying to find a good way to catch a timeout of an RQ job, so it can be requeued after the timeout. Basically, the correct solution would provide a way (for example, an exception handler in the worker or something of the sort) to requeue the job…
Juan Carlos Coto
  • 11,900
  • 22
  • 62
  • 102
7
votes
7 answers

Is there a way to assure FIFO (first in, first out) behavior with Task Queues on GAE?

Is there a way to assure FIFO (first in, first out) behavior with Task Queues on GAE? GAE Documentation says that FIFO is one of the factors that affect task execution order, but the same documentation says that “the system's scheduling may 'jump'…
André Salvati
  • 504
  • 6
  • 17
7
votes
2 answers

How to force COMMIT inside function so other sessions can see updated row?

In a Postgres 12 database, I have multiple queries (SELECT, UPDATE, ...) in a function that all together take about 20 minutes to complete. I have a check at the top that does an UPDATE if status is not running: create or replace function…
KofiB
  • 73
  • 1
  • 1
  • 4
7
votes
2 answers

How should I use Celery when task results are large?

What's the best way to handle tasks executed in Celery where the result is large? I'm thinking of things like table dumps and the like, where I might be returning data in the hundreds of megabytes. I'm thinking that the naive approach of cramming…
Chris R
  • 17,546
  • 23
  • 105
  • 172
7
votes
1 answer

Is it possible to use celery for synchronous tasks?

Nearly synchronous works, too; basically, I want to delegate the data access and processing behind a web app to a task queue for most jobs. What's the fastest latency that I can consider reasonable for celery tasks? Update (for clarification) I…
Chris R
  • 17,546
  • 23
  • 105
  • 172
7
votes
1 answer

django-celery without an amqp server (rabbitmq)

I am using celery for distributed task processing. I wanted to deploy my work on a web-host, just to show the working of my project. So how can i get djcelery to use a database (sqlalchemy) as backend with django instead of rabbitmq or other amqp…
crodjer
  • 13,384
  • 9
  • 38
  • 52
7
votes
3 answers

Networkx as a task queue?

I have a directed acyclic graph in networkx. Each node represents a task and a nodes' predecessors are task dependencies (a given task cannot execute until its' dependencies have executed). I'd like to 'execute' the graph in an asynchronous task…
jramm
  • 6,415
  • 4
  • 34
  • 73
7
votes
1 answer

Why do processes running on Google App Engine hang?

We have a cron job that runs every hour on a backend module and creates tasks. The cron job runs queries on the Cloud SQL database, and the tasks make HTTP calls to other servers and also update the database. Normally they run great, even when…
Avital
  • 91
  • 8
1 2
3
…
50 51