How do I continuously poll a rate-limited external API with Ruby on Rails?

Question

I want to query a rate-limited external API and save the information to the database. I want to get as close to the limit as possible so the information is up to date. My initial idea was to use sidekiq and create a job that was in a loop that checks for how many times the external API had been called in the past 1 second. This does not work well. I cant find any examples online. Most of the results were geared towards making an API in Ruby on Rails.

#config/initializers/data_poller.rb
require "redis"
redis = Redis.new
external_api = ExternalAPI::TestAPI1.new()
APIWorker.perform_async(external_api)


# app/workers/api_worker.rb
class APIWorker
  include Sidekiq::Worker
  def perform(external_api)
    while true
      if can_call_external_api
        external_api.get_data
      end
    end
  end
end

I don’t know much about exactly what you’re doing (what API, what kind of data, etc) but you can look into using ETags, which allows you to see if data has changed since the last time you made a call. https://en.m.wikipedia.org/wiki/HTTP_ETag — inveterateliterate, Dec 29 '17 at 03:29

score 0 · Answer 1 · answered Dec 29 '17 at 06:02

Since you are using redis, one solution would be to create a cache variable due to expire in one second.

Before the API call, check if the variable exists. If it does, do nothing. If it does not, create it again with one second expiry and do the call (or first do the call and then populate the variable)

How do I continuously poll a rate-limited external API with Ruby on Rails?

1 Answers1