How to improve async node fetch with a synchronous internet connection?

Question

I have a fast synchronous fibre connection at home. The speed is great for streams and large packages. However multiple async node fetch are very slow due to the connection overhead. I never have more than 2 async fetch from my localhost. With the connection overhead this takes roughly 1 second for every two fetches. I need more than a minute to process about 100 fetches async.

Via my 4g phone as hotspot this takes less than 2 seconds.

Is there any way to bundle fetches for synchronous internet connections?

I run this test case with node 14.

const fetch = require('node-fetch')

const promises = []

for(var i = 0; i < 100; i++) {
  promises.push(fetch('https://geolytix.github.io/public/geolytix.svg'))
}

console.time('promise all')

Promise
  .all(promises)
  .then(arr => {

    console.log(arr.length)

    console.timeEnd('promise all')
  })
  .catch(error => {
    console.error(error)
  })

10 fetch over 4g take 0.2 seconds, 100 take 1 second.

Over my gigabit line 10 fetch requests take 4 seconds, and 100 take 50 seconds.

The bevaviour with axios.get() is exactly the same.

score 1 · Accepted Answer · answered May 25 '21 at 15:50

I was able to resolve this by using a custom user agent for node-fetch. The custom user agent keeps alive and has 1 maxSockets. Increasing the maxSockets will affect the performance with synchronous internet connections.

const https = require('https');

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 1
})

const options = {
    agent: httpsAgent
}

const getPromise = () => new Promise(resolve=>{

  fetch('https://geolytix.github.io/public/geolytix.svg', options)
    .then(response => response.text())
    //.then(text => console.log(text))
    .then(() => resolve())

})

How to improve async node fetch with a synchronous internet connection?

1 Answers1