I've written a program in Node and Express, using Request to connect to an API and downloads a bunch of data (think 3,000 API requests) (all within the usage limits of the API, mind you).
When running this in a Docker container, I'm getting a lot of getaddrinfo ENOTFOUND
errors, and I'm wondering if this is a resourcing issue. My requests are like so:
request.get(url, function(err, resp, body){
// do stuff with the body here,
// like create an object and handball to a worker function
});
For the first few hundred requests this always works fine, but then I get lots nad lots of either ENOTFOUND
or timeout errors, and I think the issue might be the way my code is dealing with all these requests.
I've batched them in a queue with timeouts so the requests happen relatively slowly, it helps a little bit but doesn't solve the problem completely.
Do I need to destroy the body/response objects to free up memory or something?