Chunking axios.get requests with a 1 second delay per chunk - presently getting 429 error

Question

I have a script using axios that hits an API with a limit of 5 requests per second. At present my request array length is 72 and will grow over time. I receive an abundance of 429 errors. The responses per endpoint change with each run of the script; ex: url1 on iteration1 returns 429, then url1 on iteration2 returns 200, url1 on iteration3 returns 200, url1 on iteration4 returns 429.

Admittedly my understanding of async/await and promises are spotty at best. What I understand:

I can have multiple axios.get running because of async. The variable I set in my main that uses the async function can include the await to ensure all requests have processed before continuing the script.
Promise.all can run multiple axios.gets but, if a single request fails the chain breaks and no more requests will run.
Because the API will only accept 5 requests per second I have to chunk my axios.get requests to 5 endpoints, wait for those to finish processing before sending the next chunk of 5.
setTimeout will assign a time limit to a single request, once the time is up the request is done and will not be sent again no matter the return being other than 200.
setInterval will assign a time limit but it will send the request again after time's up and keep requesting until it receives a 200.

async function main() {
    var endpoints = makeEndpoints(boards, whiteList); //returns an array of string API endpoints ['www.url1.com', 'www.url2.com', ...]
    var events = await getData(endpoints);
    ...
}

The getData() has seen many iterations in attempt to correct the 429's. Here are a few:

// will return the 200's sometimes and not others, I believe it's the timeout but that won't attempt the hit a failed url (as I understand it)
async function getData(endpoints) {
    let events = [];
    for (x = 0; x < endpoints.length; x++) {
        try {
            let response = await axios.get(endpoints[x], {timeout: 2000});
            if (    response.status == 200 && 
                        response.data.hasOwnProperty('_embedded') && 
                        response.data._embedded.hasOwnProperty('events')
                    ) {
                let eventsArr = response.data._embedded.events;
                eventsArr.forEach(event => {
                    events.push(event)
                });
            }
        } catch (error) {
            console.log(error);
        }
    }
    return events;
}

// returns a great many 429 errors via the setInterval, as I understand this function sets a delay of N seconds before attempting the next call
async function getData(endpoints) {
    let data = [];
    let promises = [];
    endpoints.forEach((url) => {
        promises.push(
            axios.get(url)
        )
    })
    setInterval(function() {
    for (i = 0; i < promises.length; i += 5) {
        let requestArr = promises.slice(i, i + 5);
            axios.all(requestArr)
            .then(axios.spread((...res) => {
                console.log(res);
            }))
            .catch(err => {
                console.log(err);
            })
        }
    }, 2000)
}

// Here I hoped Promise.all would allow each request to do its thing and return the data, but after further reading I found that if a single request fails the rest will fail in the Promise.all
async getData(endpoints) {
    try {
        const res = await Promise.all(endpoints.map(url => axios.get(url))).catch(err => {});
    } catch {
        throw Error("Promise failed");
    }
    return res;
}

// Returns so many 429 and only 3/4 data I know to expect
async function getData(endpoints) {
    const someFunction = () => {
            return new Promise(resolve => {
                    setTimeout(() => resolve('222'), 100)
            })
    }

    const requestArr = endpoints.map(async data => {
            let waitForThisData = await someFunction(data);
            return axios.get(data)
                            .then(response => { console.log(response.data)})
                            .catch(error => console.log(error.toString()))
    });

    Promise.all(requestArr).then(() => {
            console.log('resolved promise.all')
    })
}

// Seems to get close to solving but once an error is it that Promise.all stops processing endpoint
async function getData(endpoints) {
    (async () => {
        try {
            const allResponses = await Promise.all(
                endpoints.map(url => axios.get(url).then(res => console.log(res.data)))
            );
            console.log(allResponses[0]);
        } catch(e) {
            console.log(e);
            // handle errors
        }
    })();
}

It seems like I have so many relevant pieces but I cannot connect them in an efficient and working model. Perhaps axios has something completely unknown to me? I've also tried using blurbird concurrent to limit the request to 5 per attempt but that still returned the 429 from axios.

I've been starring at this for days and with so much new information swirling in my head I'm at a loss as to how to send 5 requests per second, await the response, then send another set of 5 requests to the API.

Guidance/links/ways to improve upon the question would be much appreciated.

Chunking axios.get requests with a 1 second delay per chunk - presently getting 429 error

0 Answers0