GDAX API - Rate limit exceeded

Question

I get an error message saying "Rate limit exceeded" when I try to request historical data from GDAX. I use promises and setInterval to request historical price data from GDAX like this:

let promiseArray = //Array with promises
let j = 0;
let grabPrices = setInterval(() => {
    if (j === promiseArray.length) {
        //Do something when all promises have been resolved
    } 
    else {
        promiseArray[j].then(
            data => {
                //Work with data
            }
        ).catch(error => console.log('An error occurred: ' + error));
        j++;
    }
}, 1000) //I have even tried to use 10 seconds (10000) as a second argument to setInterval, but it doesn't make any difference.

FROM the official API documentation

Rate Limits When a rate limit is exceeded, a status of 429 Too Many Requests will be returned.

REST API PUBLIC ENDPOINTS We throttle public endpoints by IP: 3 requests per second, up to 6 requests per second in bursts.

HMR · Answer 1 · 2018-02-04T04:18:44.373

When you have a promise then the request is already made so your promiseArray is an array of ongoing requests.

Let's say I have an array of urls and use fetch to get the content. Using map to map the url to a promise and give the array of promises to Promise.all:

Promise.all(
  urls.map(fetch)
).then(
  resulst=>...
);

If urls has 10 items this program will make 10 requests immediately.

You can pass the fetch function to a throttle function that will schedule fetch in such a way that it will only be called 3 times a second. The throttle function will return a promise but will not call fetch immediately.

The throttlePeriod function can be found here. If you're only going to copy paste code from that module and not import the whole module then you need the later function as well because throttlePeriod depends on it.

const fetchMaxThreePerSecond = throttlePeriod(3,1100)/*give it an extra 100*/(fetch)
Promise.all(
  urls.map(fetchMaxThreePerSecond)
).then(
  resulst=>...
);

That takes care of the throttling but if you know how Promise.all works you know that all promises reject if only one rejects. So if you have a hundred urls and 99 resolve but one rejects your .then never gets called. You will loose the 99 successful requests.

You could pass a promise to Promise.all that will not reject, you can do so by catching a rejected promise and in the catch return a special value that you can filter out when processing results:

const fetchMaxThreePerSecond = throttlePeriod(3,1100)/*give it an extra 100*/(fetch);
const Fail = function(reason){this.reason = reason;};
const isFail = x=>(x&&x.constructor)===Fail;
const isNotFail = x=>!isFail(x);
Promise.all(
  urls.map(
    url=>
      fetchMaxThreePerSecond(url)
      .catch(e=>new Fail([url,e]))//save url and error in Fail object
  )
)
.then(//even if a request fails, it will not reject
  results=>{
    const successes = results.filter(isNotFail);
    const failed = results.filter(isFail);
    if(failed.length!==0){
      console.log("some requests failed:");
      failed.forEach(
        (fail)=>{
          const [url,error] = fail.reason;
          console.log(`Url: ${url}`);
          console.log(JSON.stringify(error,undefined,2));
        }
      )
    }
  }
);

score 0 · Answer 2 · answered Feb 17 '18 at 17:37

I have no problem with GDAX history calls when I set my rate limiter to 3 API calls every 2.1 seconds.

I have some occasional problems when I set my rate limiter to 3 API calls every 1.8 seconds.

I have tested these values with automated tests.

Important: all my calls to GDAX share the same rate limiter(!)

To be save, use the code from @HMR's answer together with my tested parameters.

GDAX API - Rate limit exceeded

2 Answers2

Linked