0

I am working on a project where I need to get related data from a server for 1 million different data points. If I try to make concurrent requests using a loop and make an array of promises and resolve the promise when I have received all the data through Promise.all I get an error ERR_INSUFFICIENT_RESOURCES.

{
  ...
  var arr = [];
  for(var i = 0; i < data.length; i++){
    arr.push(getData(data[i]);
  }

  Promise.all(arr).then(doSomething);
 
}

function getData(data) {
  var url = '/url&data=' + data;
  return fetch('url');
}

How can I get all the data from the server for all the data points in the least amount of time and without running into errors? I do not wish to make serial requests as that would take a very long time.

Shadab Khan
  • 97
  • 1
  • 2
  • 8
  • 4
    Are you planning for DoS attack? – Satpal May 12 '17 at 08:45
  • 1
    1 million request from a single client ? – abhishekkannojia May 12 '17 at 08:45
  • @satpal True :) By the way. What is the amount of requests that can be handled by your client? 100? 1000? When browser is not crushing? – Sharikov Vladislav May 12 '17 at 08:46
  • 1
    seems like a typical speed vs memory choice, consider that most of the time you can't have both. If not enough resources, it will take more time, and probably require sequencial requests – Kaddath May 12 '17 at 08:48
  • I think you need to use some kind of limited thread pool. Like Satpal says, you're just DOSing your own service. – HomerPlata May 12 '17 at 08:48
  • @Satpal: no, trying to find a way to make a project scalable to be able to take a large amount of data from server and then get related data for each data point. – Shadab Khan May 12 '17 at 08:48
  • @Sharikov: around 5000 at max – Shadab Khan May 12 '17 at 08:49
  • @Kaddath: exactly the problem I am facing. Thanks for putting it into more specific terms :-) – Shadab Khan May 12 '17 at 08:51
  • One way that I can think of is sequentially making batches of requests. Say making a mx of 2000 requests. – Shadab Khan May 12 '17 at 09:02
  • 3
    If you're trying to batch a lot of data points within a single request, that's something you'll need to implement in your server, and I don't know if we'll be able to help further without knowing a lot of your server implementation details. If you want to keep a large number of requests going at once without doing a million simultaneously, [that's a throttling problem](http://stackoverflow.com/q/38385419/1426891). – Jeff Bowman May 12 '17 at 16:38
  • "... take a large amount of data from server and then get related data for each data point". Consider, if possible, refactoring your request architecture to avoid the client having to orchestrate the fetching of related data. Have one request type and write your server resource to return basic + related data from each request. – Roamer-1888 May 12 '17 at 23:46
  • @JeffBowman the link you provided is something that I can consider using. Thanks :-) – Shadab Khan May 13 '17 at 06:01

0 Answers0