0

How would I structure a curl command to have it recognized by a URL shortener? When I run the command below using a url shortened from either bit.ly or goo.gl, the request is not recognized by either service.

curl -L "http://bit.ly/1efYimy"
adayzdone
  • 11,120
  • 2
  • 20
  • 37
  • I am trying to get the request to register as a "click" in the URL shortening service. – adayzdone Feb 25 '14 at 17:53
  • Your url is redirected to `https` link. use `-k` parameter as well. To debug the thing, use the `-v` for that. – Sabuj Hassan Feb 25 '14 at 18:00
  • With or without the -k, I get the correct response. However, the request is not being registered in the URL shortener. – adayzdone Feb 25 '14 at 18:25
  • I don't know how the shortener site detect the call. Final request. Add `-A "Mozilla/5.0 (Windows NT 5.1)"` so that the site gets a hint that you have requested it from a browser! – Sabuj Hassan Feb 25 '14 at 18:36
  • That doesn't seem to register either... – adayzdone Feb 25 '14 at 18:44
  • `curl -L "http://bit.ly/1efYimy"` works perfectly for me. Would you explain what you mean by `not recognized by either service`? – Samveen Feb 25 '14 at 19:05
  • @Samveen If you created a shortened URL in one of those services and then issue the curl -L command pointing towards the shortened link, although the response is correct, the URL shortener is not recording a "click". Try it yourself. – adayzdone Feb 25 '14 at 19:08
  • @adayzdone So the request goes to the URL shortener, but doesn't show up in the URL shortener's analytics, is it? – Samveen Feb 26 '14 at 06:46

1 Answers1

1

Most URL shorteners do various forms of filtering and normalization to exclude "bot" traffic from the statics they report.

If you are making a low volume of requests, you can probably get your requests counted by changing the User Agent reported by curl.

curl -LA "MyApp 1.0" "http://bit.ly/1efYimy"

or

curl -L --user-agent "MyApp 1.0" "http://bit.ly/1efYimy"

Please be a good web citizen and pick a user agent that describes what you are doing and has some reference to how you can be found/contacted (e.g. "MyCo Bot" where doing a Google search for MyCo will bring somebody to your company).

Also note that while user agent based filtering is one form of bot detection that is done, there are more sophisticated measures that various services will use to detect bot-like behavior. Accordingly, if you end up making a large volume of requests your metrics may stop incrementing or you may even get blocked/rate limited.

SeanOC
  • 1,331
  • 1
  • 8
  • 12