Delaying node.js HTTP requests

Question

I have created a Node.js script that reads data from a CSV file, and then makes an axios POST request to an endpoint to create a user in a database. My issue is, when working with a lot of data (100+ users in CSV) I am sending so many requests at one time that I am getting a 429 status code that says too many requests. Is there any way to space these out?

Here is my code :

const csv = require('csv-parser');
const fs = require('fs');
const axios = require('axios');

let dataArray = [];

let API_KEY = 'example key'

fs.createReadStream('output.csv')
  .pipe(csv())
  .on('data', async (row) => {
      try {
    let hluser = JSON.stringify({"TenantCode":"","Items":[{"MemberDetails":{"LegacyContactKey": row.hl_uid,"MemberID":"","EmailAddress": row.email,"FirstName": row.first_name,"LastName": row.last_name,"CompanyName": row.Company,"Title": row.job_title,"Phone1":"","Phone1Type":"Phone","Phone2":"","Phone2Type":"Office","Address1":"","City": row.city,"State": row.state,"PostalCode":"","Country": row.country,"Bio":"","ProfileImageURL":"","FacebookURL":"","TwitterURL":"","LinkedInURL":"","WebsiteUrl":"","IsMember":"True","IsActive":"True"},"CommunityGroups":[{"GroupKey":"","RoleDescription":"Member"}]}]});

    console.log(hluser);
    let config = {
      method: 'POST',
      url: 'example.com/v1/api',
      headers: { 
        'x-api-key': API_KEY, 
        'Content-Type': 'application/json'
      },
     data: hluser
    };
    
    const {data} = await axios(config)
    console.log(data);
    console.log('***********************************');
    console.log(`Successfully added ${row.first_name} ${row.last_name} to Higher Logic`);
    console.log('***********************************');

    console.log(row);
  } catch (err) {
      console.log(err);
      console.log(`Could not add ${row.first_name} ${row.last_name}`);
  }})
  .on('end', () => {
      
    console.log('CSV file successfully processed');
  });

You need a throttle mechanism! Does this answer your question? https://stackoverflow.com/q/32785484/5605822 — Tasos, Oct 26 '20 at 17:03
Wouldn't it be better to [batch the Axios](https://stackoverflow.com/a/61588558/1563833) requests? — Wyck, Oct 26 '20 at 17:04

score 0 · Answer 1 · answered Oct 26 '20 at 17:43

There's a nice library called axios-rate-limit that you can use to rate-limit your requests.

So if you want to limit your requests to e.g. 5 requests per second you can simply do:

const http = rateLimit(axios.create(), { maxRequests: 5, perMilliseconds: 1000 });

fs.createReadStream('output.csv')
  .pipe(csv())
  .on('data', async (row) => {
   // ...
   const {data} = await http(config)
  // ..
  });

Delaying node.js HTTP requests

1 Answers1