0

I need to use a queue to serialize my async requests. Looking around, I've found a small library by Mike Bostock here. But I am a bit confused as tp how to use it along with promise object.

So, I have tons of reqs coming from the user interface. 

function addTask(d){
   AsyncOper(d)
      .then(function () {
             refresh()
      });
}

AsyncOper returns a promise object (angular js implementation - $q).

I have a q defined as

var q = queue(1);

How can I turn addTask to use q

My first attempt was as follows:

    function addTask(d){
       q.defer(request, d)
       q.awaitAll(function(error, results) { console.log("all done!"); });
    }
    function request(d, cb) {
        AsyncOper(d)
            .then(function () {
                refresh();
                cb(null, "finished "+ d);
            })
    }

But it is not really serializing the operation, since I see it trying to run more than one request. Is it possible to combine promise and queue this way or is there a better way?.

Thank you.

RAS
  • 8,100
  • 16
  • 64
  • 86
bsr
  • 57,282
  • 86
  • 216
  • 316
  • 1
    This should work. As @Iain pointed out, the docs warn against calling `awaitAll` repeatedly, but even that (at least in the current version of the library) should not keep this from working. Seems like it must be something else. Note that if you just want to keep a continual queue, you do not need to call `awaitAll`, the queue will start processing jobs immediately. – numbers1311407 May 04 '13 at 04:38
  • +1 thanks. I tried without awaitAll, and it still works. I don't care about final result after task finished, and hope there is no harm doing this way. Thanks again – bsr May 04 '13 at 13:00
  • This is a pretty cool little lib but I do think it's suffering from an identity crisis (is it a queue, or a serial/parallel task runner?). This probably isn't an issue but it should be noted, that if used as indefinitely running queue it has a bit of a built in memory leak, as the results of the jobs are stored forever, waiting for an `await(All)`. – numbers1311407 May 04 '13 at 15:49
  • Also, even though it's named "queue-async", this lib itself *is not* async but rather depends on the jobs to be. If you queue a sync job, your script will block on it. Wouldn't be an issue for you here, but worth noting. – numbers1311407 May 04 '13 at 15:54
  • 1
    thanks again for mentioning the memory leak issue.Do you have any recommendation for a tiny lib, with only one usecase - queueing sync task which need to process one at a time. Mainly limit the rate at which it can run. – bsr May 04 '13 at 15:58
  • 1
    Nothing comes to mind really. This lib you're using now is pretty close to as tiny as you're gonna get. If your jobs aren't returning data, the queue will only maintain an unbound sparse array, which isn't much of a memory leak (and depending on the browser implementation, maybe not one at all if space is not allocated for undefined index). But if you wanted to modify the lib a little it would be trivial to replace the job array with a simple object, which you could delete the job key from on completion. As long as you respect the license it should be fine to modify and re-use the code. – numbers1311407 May 04 '13 at 16:25
  • Check this [Queue class](https://stackoverflow.com/questions/38903707/nodejs-create-simple-queue-on-async/51482852#51482852) I wrote for this purpose. – Shl Aug 05 '18 at 09:55

1 Answers1

3

https://github.com/mbostock/queue says, about await and awaitAll:

This method should only be called once, after any tasks have been
deferred. If the await callback is set multiple times, or set before 
a task is deferred, the behavior of the queue is undefined.

At the moment, you call awaitAll after every time you defer a task, which is some definite dragon-tickling. Is there a place you can all awaitAll after queueing everything? If not, this particularly library may not be right for you.

Iain
  • 4,203
  • 23
  • 21
  • I read that bit as well, but looking at [the source](https://github.com/mbostock/queue/blob/master/queue.js#L69) it seems that all the `await` functions do is overwrite the callback executed when the queue drains, and call it if the queue is already empty. I'm not immediately seeing why the OP's code does not work. Any thoughts? – numbers1311407 May 04 '13 at 03:43
  • You're right, it's not obvious from the source why it shouldn't work. I just have a philosophical objection to doing stuff that the documentation says you shouldn't do and hoped that in this case that objection might translate to a bugfix. The next questions I would ask are: what does refresh() do? have you proven that AsyncOper() doesn't resolve until the request is actually finished? Is there only one assignment to `q`, which definitely has a parallelism of 1? – Iain May 04 '13 at 05:01
  • Thanks @numbers1311407. your comments really helped. My AsyncOper was a series of (chain) of async operation, and I was not serializing one of them. Once I serialize all the operation in the chain, it works fine. Thanks again for the help. – bsr May 04 '13 at 13:03