Add limit/delay on fetch requests to an API

Question

I would like to prevent the API I'm using from throwing me 502 Bad Gateway. At the moment I'm looping through the array of web URL's and requesting their screenshot. The issue is I do too many requests per second. I would like to convert my code so there is a queue and requests are landing every X amount of time (eg. every 5 sec). This way server won't block my requests.

Here is my example code:

https://jsfiddle.net/fqu2o7zd/17/

const urls = ['app.xrespond.com','sizzy.co','www.heidisql.com','quirktools.com','regex101.com','lab.hakim.se','anthonyterrien.com','casperin.github.io','joaopereirawd.github.io','humaan.com','dimsemenov.com','compressor.io','jsbin.com','import.io','ecomfe.github.io','codeanywhere.com','ceagon.com','www.chartjs.org','c3js.org','www.fusioncharts.com','omnipotent.net','www.cloudnotic.com','github.hubspot.com','jquerygrid.net','razorjack.net','select2.github.io','www.virtuosoft.eu','nicolasbize.com','bxslider.com','compressor.io','jsbin.com','import.io','ecomfe.github.io','codeanywhere.com','ceagon.com','www.skitter-slider.net','jedrzejchalubek.com','luis-almeida.github.io','www.thepetedesign.com','www.slidesjs.com','fancyapps.com','www.chartjs.org','c3js.org','www.fusioncharts.com','omnipotent.net','www.cloudnotic.com','github.hubspot.com','bitbrewery.de','jquerygrid.net','razorjack.net','select2.github.io','www.virtuosoft.eu','nicolasbize.com','bxslider.com','www.skitter-slider.net','jedrzejchalubek.com','luis-almeida.github.io','www.thepetedesign.com','fancyapps.com','bitbrewery.de','www.slidesjs.com','compressor.io','www.skitter-slider.net','compressor.io','www.thepetedesign.com','www.fusioncharts.com','www.skitter-slider.net','fancyapps.com','codeanywhere.com','www.fusioncharts.com','www.thepetedesign.com','compressor.io','www.fusioncharts.com','www.thepetedesign.com','fancyapps.com','codeanywhere.com','fancyapps.com','codeanywhere.com','luis-almeida.github.io','www.chartjs.org','ceagon.com','www.cloudnotic.com','razorjack.net','jquerygrid.net','omnipotent.net','luis-almeida.github.io','ceagon.com','www.chartjs.org','razorjack.net','jquerygrid.net','www.cloudnotic.com','github.hubspot.com','ecomfe.github.io','bxslider.com','nicolasbize.com','jsbin.com','www.skitter-slider.net','luis-almeida.github.io','ceagon.com','omnipotent.net','jedrzejchalubek.com','www.chartjs.org','github.hubspot.com','import.io','bxslider.com','ecomfe.github.io','select2.github.io','jsbin.com','c3js.org','nicolasbize.com','bitbrewery.de','omnipotent.net','jedrzejchalubek.com','www.virtuosoft.eu','github.hubspot.com','bxslider.com','select2.github.io','import.io','www.slidesjs.com','c3js.org','www.cloudnotic.com','bitbrewery.de','ecomfe.github.io','compressor.io','razorjack.net','jsbin.com','nicolasbize.com','www.fusioncharts.com','www.slidesjs.com','www.thepetedesign.com','codeanywhere.com','jedrzejchalubek.com','select2.github.io','jquerygrid.net','fancyapps.com','c3js.org','bitbrewery.de','www.skitter-slider.net','ceagon.com','luis-almeida.github.io','www.slidesjs.com','www.chartjs.org','import.io','omnipotent.net','github.hubspot.com','bxslider.com','www.cloudnotic.com','ecomfe.github.io','razorjack.net','jsbin.com','nicolasbize.com','jedrzejchalubek.com','select2.github.io','jquerygrid.net','c3js.org','bitbrewery.de','www.slidesjs.com','import.io','codeanywhere.com','compressor.io','www.chartjs.org','compressor.io','www.chartjs.org','omnipotent.net','compressor.io','github.hubspot.com','www.chartjs.org','github.hubspot.com','bxslider.com','bxslider.com','www.cloudnotic.com','github.hubspot.com','omnipotent.net','ecomfe.github.io','www.cloudnotic.com','bxslider.com','ecomfe.github.io','razorjack.net','omnipotent.net','www.cloudnotic.com','razorjack.net','jsbin.com','ecomfe.github.io','jsbin.com','nicolasbize.com','razorjack.net','nicolasbize.com','jedrzejchalubek.com','jsbin.com','jedrzejchalubek.com','select2.github.io','nicolasbize.com','select2.github.io','jquerygrid.net','jedrzejchalubek.com','jquerygrid.net','c3js.org','select2.github.io','c3js.org','bitbrewery.de','jquerygrid.net','bitbrewery.de','www.slidesjs.com','c3js.org','www.slidesjs.com','import.io','bitbrewery.de','import.io','compressor.io','www.slidesjs.com','compressor.io','www.chartjs.org','import.io','www.chartjs.org','github.hubspot.com','compressor.io','github.hubspot.com','bxslider.com','www.chartjs.org','bxslider.com','omnipotent.net','github.hubspot.com','omnipotent.net','www.cloudnotic.com','bxslider.com','www.cloudnotic.com','ecomfe.github.io','omnipotent.net','ecomfe.github.io','razorjack.net','www.cloudnotic.com','jsbin.com','ecomfe.github.io','nicolasbize.com','jsbin.com','jedrzejchalubek.com','razorjack.net','jedrzejchalubek.com','nicolasbize.com','jsbin.com','razorjack.net','jedrzejchalubek.com']


function _arrayBufferToBase64(buffer) {
    var binary = ''
    var bytes = new Uint8Array(buffer)
    var len = bytes.byteLength
    for (var i = 0; i < len; i++) {
        binary += String.fromCharCode(bytes[i])
    }
    return window.btoa(binary)
}

function _createDiv(data) {

  const id = '_' + Math.random().toString(36).substr(2, 9)
  const div = document.createElement('div')
 div.id = id
  document.body.appendChild(div)
  document.getElementById(id).innerHTML = `<img src="${data}" width="300">`
  
}

/* Helpers Above ################################################################ */

/* Fetching from API */
function fetchImg (url) {

    const requestAPI = new Request(`https://mini.s-shot.ru/?http://${url}`)
    const options = {
        method: 'GET',
        mode: 'cors',
        cache: 'default'
    }
    
    fetch(requestAPI, options).then((response) => {

        response.arrayBuffer().then((buffer) => {

            const resContent = response.headers.get('content-type')
            const resStatus = response.status
            const image64 = `data:${resContent};base64,` + _arrayBufferToBase64(buffer)

            if (resStatus === 200) {

              _createDiv(image64)

            } else {

              console.log(`Fetch Issue: ${resStatus}`)

            }

        })
    })

}

/* URL Loop */
urls.forEach((url) => {
  
  fetchImg(url)
  
})

do you realise that most, if not all, of those requests fail due to CORS - slowing down requests won't fix your issue — Jaromanda X, Apr 22 '19 at 09:46
This is one example and I’m aware for CORS. My question is how to slow down fetch requests. — Ben, Apr 22 '19 at 10:57

score 1 · Answer 1 · answered Apr 22 '19 at 12:41

If you need to delay between requests, then you'll need to

chain the promises
put a delay between requests

So, with that in mind, here's a simple delay you can use with promises

const delay = ms => new Promise(r => setTimeout(r, ms));

And then you'll change your code to return the promises from your function

function fetchImg (url) {
    const requestAPI = new Request(`https://mini.s-shot.ru/?http://${url}`)
    const options = {
        method: 'GET',
        mode: 'cors',
        cache: 'default'
    }
    // RETURN HERE
    return fetch(requestAPI, options).then((response) => {
        // AND HERE
        return response.arrayBuffer().then((buffer) => {

            const resContent = response.headers.get('content-type')
            const resStatus = response.status
            const image64 = `data:${resContent};base64,` + _arrayBufferToBase64(buffer)

            if (resStatus === 200) {

              _createDiv(image64)

            } else {

              console.log(`Fetch Issue: ${resStatus}`)

            }

        })
    })

}

Now, to use the functions above

urls.reduce((p, url) => p.then(() => delay(10)).then(() => fetchImg(url)), Promise.resolve());

Or (inside an async function) far cleaner code ...

for (let url of urls) {
    await fetchImg(url);
    await delay(10);
}

Hi @Jaromanda X, thanks for your help. In your example, we are putting a delay on the loop itself (reduce) which slows down rendering other items. I did what I'm looking for based on two separate loops (faster & slower) here: https://jsfiddle.net/nLtz2e7x/5/ Is there any better way combining this into one? Thanks — Ben, Apr 22 '19 at 21:11

Add limit/delay on fetch requests to an API

1 Answers1