0

I am running a Rails 3 app with ruby 1.9.3-p448 and using the Curl::Easy library to scrape some images when a user includes a url in a post.

lets set:

url_a = "http://hardballtalk.nbcsports.com/2014/04/07/team-exec-thinks-shortening-games-to-seven-innings-is-what-baseball-needs/"

url_b = "http://www.cbssports.com/golf/eye-on-golf/24563578/us-open-infographic-shows-path-to-championship"

When I use the user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0' I find that curling url_a works fine but url_b returns a compressed response.

When I use the user-agent 'Mozilla/5.0' the above url_a yields this error:

Curl::Err::GotNothingError: Curl::Err::GotNothingError

while curling url_b gives an uncompressed response (which is what I desire).

Anyone know why this is happening and what would be a good user-agent choice to be able to return good, uncompressed responses from both urls?

I'm a bit out of my comfort zone here, so please ask me for any more information you need.

Thanks!

wuliwong
  • 4,238
  • 9
  • 41
  • 69

1 Answers1

0

This works for me all the time:

"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0"
Kolkata Calcutta
  • 59
  • 9