Why is Array.push(element) not working sometimes?

Question

I'm using node.js for a project, and I have this certain structure in my code which is causing problems. I have an array dateArr of sequential dates that contains 106 items. I have an array resultArr to hold resulting data. My code structure is like this:

function grabData(value, index, dateArr) {
  cassandra client execute query with value from dateArr {
    if (!err) {
      if (result has more than 0 rows) {
        process the query data
        push to resultArr
      }
      if (result is empty) {
        push empty set to resultArr
      }
    }
  }
}

dateArr.forEach(grabData);

I logged the size of resultArr after each iteration and it appears that on some iterations nothing is being pushed to resultArr. The code completes with only 66 items stored in resultArr when 106 items should be stored because the I/O structure between dateArr and resultArr is 1 to 1.

I'm pretty sure `cassandra client execute query with value from dateArr` is asynchronous. — Felix Kling, Jun 25 '15 at 18:22

score 0 · Answer 1 · answered Jun 25 '15 at 18:40

0

I logged the size of resultArr after each iteration

When the grabData method gets called you start a query to somewhere, or someone named cassandra. As Felix Kling wrote, your notation seems to show an asynchronous function, that starts the request and returns.

As the function is asynchronous, you don't know when the query is ready. That might even take very long, when the database is locked for a dump, or whatever.

When you return from grabData "iteration" and check your resultArr, the resultArr will exactly be filled with each returned value. It might even be that the fifth iteration returns a query before the third, or fourth or any iteration before. So in you resultArr you sometimes have values of iteration n at some point m<n or o>n.

As long as you (or we) don't know anything about how cassandra operates, you cannot say when a query gets answered.

So when you check your result array, it returns the number of completed queries, not the number of iterations.

answered Jun 25 '15 at 18:40

ikrabbe

1,909
12
25

Is there a way to reduce the time that it takes for each iteration to return a completed query? – hologram Jun 25 '15 at 18:46
You can reduce it as much as you want. If cassandra is slow, it will never get handled before grabData returns. Of course when cassandra is just a javascript proxy function returning a constant value, that speed might be enough. But that still depends on the switching of parallel context in the javascript vm. Anyway, a proxy function with constant data won't return much interesting data. – ikrabbe Jun 25 '15 at 18:48
True - but, is there a best practice here? – hologram Jun 25 '15 at 18:49
No, the best practice with asynchronous results is to think of them as something that happens some time in the future. Noone really knows as there might be many external factors. To see what happens add some logging inside of the return function, best behind the `if(!err)` block. Of course giving speed to cassandra by scaling her up into several parallel high speed nodes generally helps. But that's some kind of brute force. – ikrabbe Jun 25 '15 at 18:53

score 0 · Answer 2 · answered Jun 25 '15 at 20:53

0

Found the root cause: There is a hard limit when querying Cassandra using node.js. The query that I am trying to completely execute is too large. Breaking dateArr up into smaller chunks and querying using those smaller pieces solved the problem.

answered Jun 25 '15 at 20:53

hologram

533
2
5
21

Why is Array.push(element) not working sometimes?

2 Answers2