Ruby Net:Http get request gives different response than with Browser

Question

I am trying to fetch from API server using Net::HTTP.

puts "#{uri}".green
response = Net::HTTP.new('glassdoor.com').start { |http|
  # always proxy via your.proxy.addr:8080
  response =  http.get(uri,  {'Accept' => 'application/json'})
  puts "Res val: #{response.body}".blue
}

I got the uri from the console and pasted in the browser, and I received the JSON response.

But using the Ruby Net::HTTP get I receive some security message:

Why the difference? The browser and the Ruby script are behind the same public IP.

Answer 1

You were detected as a crawler (correctly, by the way). Note that those requests (from browser and the script) are not just the same. The browser sends some headers, such as accepted language, user agent etc. You can peek into it using web inspector tool in the browser. On the other side, in your script you only set Accept header (and to JSON, suspicious on its own, as browser would never do that). And you do not send any user agent. It's easy to see that this is an automates request, not natural traffic from the browser.