0

We've about 800 json files to be posted to a Shopify API. It can't be merged into a single API call (there's no BULK post API). Shopify limits 40 requests per second. This is the API we're posting data :

POST /admin/api/2019-07/inventory_levels/set.json.

At data source we've less control over how/when the data is generated and post. But we can accumulate json files in a folder.

We're looking for a tool (or a test suite) that can allow something like - API post in chunks or a way to schedule/limit post API requests in queue/group. We've found "Scheduler" features in tools like JMeter but hope to a solution from someone who has already designed similar solutions.

Thanks in advance

Hemant Tank
  • 1,724
  • 4
  • 28
  • 56
  • This question is almost impossible to answer. Making HTTP POST calls is so common and easy, in almost every known scripting language in common use. If you cannot organize your "source" to do a POST, you must reveal why for help. – David Lazar Aug 28 '19 at 00:39
  • @DavidLazar we're trying to handle it on source but we haven't been able to achieve it yet. Having a node.js like script which monitors a folder for .json file and does what we need is the last option. If a tool is available, we'd like to use it directly. – Hemant Tank Aug 28 '19 at 15:16
  • @HemantTank, can't you set up a webhook that'll receive your raw JSON data, and instead of storing it in a file, send to Shopify via API? – HymnZzy Aug 29 '19 at 07:45
  • @HemantTank, and secondly, if uploading 800 files is a one time activity you do it programmatically. This can be sequential or parallel calls as you see fit. – HymnZzy Aug 29 '19 at 07:49
  • @HymnZ - as I've mentioned we're trying another approach from within our interface. Files will be created. However, we need to know if there's a tool that can automate it - the main feature we need is scheduling to overcome the request per second limitation. Thx. – Hemant Tank Sep 14 '19 at 14:57

1 Answers1

1

The Shopify REST Admin API applies rate limits to the API requests that it receives. Every request is subject to throttling under the general limits. The rate limits are designed to allow your app to make unlimited requests at a steady rate over time while also having the capacity to make infrequent bursts. The rate limits use a leaky bucket algorithm. The bucket size and leak rate properties determine the API's burst behavior and request rate.

The default settings are as follows:

  • Bucket size: 40
  • Leak rate: 2/second

If the bucket size is exceeded, then an HTTP 429 Too Many Requests error is returned.

Recommendations

  1. Design your app with API rate limits in mind to best handle request limits and avoid 429 errors.
  2. Stagger API requests in a queue and do other processing tasks while waiting for the next queued job to run.

Use the rate limits header to balance your request volume. We handled this use case using celery. Basically, you need to make API calls and whenever a call fails you get an exception code. If the code is 429 then you retry the same task in an interval(below check how to get the interval) till it is successful.

How to get the interval after which you can make a request after your app is throttled?
If your app is throttled, then it won't be able to make any more requests until enough time has passed and your bucket has capacity again. You can calculate this wait time manually using the bucket size and leak rate properties, or by using the Retry-After response header

Source: Shopify Rate Limit Documentation

Subhrajyoti Das
  • 2,685
  • 3
  • 21
  • 36
  • 1
    Thanks, that was informative and I believe it confirms that Shopify doesn't have a bulk "set" API and we've to obey the throttle limits. Will look into Celery. – Hemant Tank Aug 28 '19 at 15:16