Optimizing HTTP Status Request on Heroku (Net:HTTP)

Question

I'm running an app on heroku where users can get the HTTP status (200,301,404, etc) of several URLs that they can paste on a form.

Although it runs fine on my local rails server, when I upload it on heroku, I cannot check more than 30 URLs (I want to check 200), as heroku time outs after 30seconds giving me an H12 Error.

  def gethttpresponse(url)
  httpstatusarray = Hash.new

    uri = URI.parse(url)
    http = Net::HTTP.new(uri.host, uri.port) 
    request = Net::HTTP::Get.new(uri.request_uri)
    response = http.request(request)
    httpstatusarray['url'] = url
    httpstatusarray['status'] = response.code               
   return httpstatusarray

  end

At the moment I'm using Net:HTTP, and it seems very heavy. Is there anything I can change on my code or any other gem I could use to get the HTTP status/headers on a more efficient (fast) way?

i noticed that response.body holds the entire HTML source code of the page which i do not need. is this loaded on the response object by default?

If this is the most efficient way to check HTTP Status, would you agree that this needs to become a background job?

Any reference on gems that work faster, reading material and thoughts are more than welcome!

Answer 1

If a request takes longer than 30 seconds then the request is timed out as you're seeing here. You're entirely dependent on how fast the server the other end responds. For example, if say it were itself a Heroku application on 1 dyno then it will take possibly 10 seconds to unidle the application therefore leaving only 20 seconds for the other URLs to be tested.

I suggest you move each check to it's own background job and then poll the jobs to know when they complete and update the UI accordingly.