buffer tasks processing

Question

The task is to process a lot of commands that needs to be saved in some kind of stack or buffer. While one method will push the data in, there will be multiple threads or processes, that will take thees tasks one by one and process them.

Right know the Idea is to save the tasks in buffer that uses NoSQL database, so we can get object and delete it simultaneously.

I'm thinking, that for this problem probably is already solution with some kind of server or library, that handles task processing and distribution between multiple instances.

Is there such a thing?

this is a classic - producer/consumer problem. look it up on the webz. — nullpotent, Feb 27 '12 at 13:06
Thank you! Is there some out of the box implementations for this or it is best to code it by pattern myself? — wormhit, Feb 27 '12 at 13:14

score 0 · Answer 1 · edited May 23 '17 at 12:17

0

I found gearman, php queue manager, and solution with nodejs node-amqp and php-amqp using RabbitMQ

edited May 23 '17 at 12:17

Community

1
1

answered Feb 27 '12 at 14:20

wormhit

3,687
37
46

score 0 · Accepted Answer · answered Feb 28 '12 at 12:17

Well the pattern implementation depends on your specific needs. Your question is too general for a better answer than the one provide as a comment by @AljoshaBre: "this is a classic - producer/consumer problem. look it up on the webz.". If you look at the wikipedia article about producer/consumer problem, you can find the pattern implementation in Java - the general pattern is small, but to address your specific needs more details are required. You say something about "task processing and distribution between multiple instances", and it leads for a more specific architectural pattern called Distributed Message Queues (some random ref). There is the Apache project ActiveMQ that aims to implement such pattern.

buffer tasks processing

2 Answers2