0

I'm using the unirest library to fetch all of the data from an api, which is split up by offset and limits parameters, and has no finite number of results.

I'm using a while condition to iterate through the data and at the point where no results are returned, I end the loop by setting an 'incomplete' variable to false.

But for some reason, when I run the following code nothing happens (as in no data is added to my database and nothing is outputted to the console) until I get the 'call_and_retry_last allocation failed' error (assuming this happens when a while loop goes on too long). But when I remove the while condition altogether the code works fine.

Is there a particular reason why this isn't working?

Here's my code:

var limit = 50,
    offset = 0,
    incomplete = true;

while (incomplete) {

    // make api call
    unirest.get("https://www.theapiurl.com")
        .header("Accept", "application/json")
        .send({ "limit": limit, "offset": offset })
        .end(function (result) {

            // parse the json response
            var data = JSON.parse(result.raw_body);

            // if there is data
            if( data .length > 0 )
            {
                // save the api data
                // + increase the offset value for next set of data
                offset += limit;
            } 
            else
            {
                // if there is no data left, end loop
                incomplete = false;
                console.log("finished!");
            }
        });
    }
mmmoustache
  • 2,273
  • 6
  • 41
  • 62
  • 1
    It should be working. But by the time, your response comes in, the loop goes further. Use Async.io or something in a waterfall or series model. – Ani Nov 20 '16 at 15:01
  • 1
    You trying to use async and sync code together. While loop sends too much requests and you get stack size error. Use http://caolan.github.io/async/ or try to create promises array and then run it by Promise.all. Hope this helps. – Mykola Borysyuk Nov 20 '16 at 15:02
  • 1
    To make your loop pause for the async action to complete, take a look at `fibers/future`, `fut.wait()` at the end of the loop's body, `fut.return()` at the end of the `.end`'s body (only possible in _nodejs_) – Paul S. Nov 20 '16 at 15:05
  • thanks for the suggestions, using the async utility solved my problem! – mmmoustache Nov 20 '16 at 15:37

1 Answers1

0

You can use recurrcive function as

function getServerData(offset){

//Your api service with callback.if there is a data then call it again with the new offset.

}
function getServerData(1);
raths
  • 31
  • 5