0

In one of my queries, I am collating ~250 calls to one array of results using Bluebird Promise.map function. Each individual call takes ~60ms when executed by itself so with a concurrency of 10 I would expect the result of the main query to be around 1.5sec (=(250/10)*60ms).

But I'm getting my answer in the ~15 seconds range as if each query was executed one after the other. No matter what value I give to my concurrency parameter I always get the same (too long) answer time.

Here is my code (I am using the nano module in a node.js server): 

console.time("total");
let concurrency = req.query.hasOwnProperty("concurrency") ? Number(req.query.concurrency) : 20;
console.time("list_dbs");
nano.db.list().then((body) => {
    console.timeEnd("list_dbs");
    let dbs = body.filter((db) => db.search(/^ts_[0-9]{4}-[0-9]{2}/g)>-1);
    console.log(dbs.length); //244
    console.time("Promises");
    Promise.map(
        dbs,
        (db_name) => {
            console.time(db_name);
            let db = nano.use(db_name);
            return db.view(
                "blah",
                "blah" ,
                {
                    group_level:1,
                    stale:"ok"
                }
            ).then((body) => {
                console.timeEnd(db_name);
                let blahs = [];
                body.rows.forEach((row) => {
                   blahs.push(row.key[0]);
                });
                return(blahs);
            }).catch((err) => {
                return err;
            });
        },
        {concurrency: concurrency}
    ).then((values) => {
        console.timeEnd("Promises");
        res.send((_.union(_.flatten(values))).sort());
        console.timeEnd("total");
    }).catch((err) => {
        res.send(err);
        console.timeEnd("total");
    });
}).catch((err) => {
    res.send(err);
    console.timeEnd("total");
});

Am I doing something wrong with my use of Promise.map? Is there a parameter in couch that is throttling the speed? How to speed up that query?

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Chapo
  • 2,563
  • 3
  • 30
  • 60
  • Something must be taking longer than you think - because I can confirm with concurrency 10, the time taken at 60ms is indeed 1504ms in simple testing, 782ms for concurrency of 20, and 68ms with no concurrency specified (i.e. full parallel) - all values pretty much as expected – Jaromanda X Apr 03 '19 at 09:45
  • 1
    perhaps there's a concurrency limit with accessing the database? You should log start end times for the db access – Jaromanda X Apr 03 '19 at 09:47
  • I've increased all parameters I could find for access to the db but I might have missed something... The code for the `Promise.map` is properly written then ? – Chapo Apr 03 '19 at 10:04
  • Have you log the database access times like @JaromandaX suggested? This will let you know if any requests are slower. – Alexis Côté Apr 03 '19 at 17:50
  • I can tell you I stripped all the db code from your code, made a dummy promise that took 60ms to resolve, and your code takes 1.5s to "process" 250 items with a concurrency of 10 - your code looks fine, it's your assumptions that are incorrect – Jaromanda X Apr 03 '19 at 22:01
  • @AlexisCôté not sure how to do that unfortunately. My `console.time` -> `console.timeEnd` are not sufficient ? – Chapo Apr 04 '19 at 01:16

0 Answers0