1

For example, I have an action which responds to an json ajax request on my site.

Recently I found a lot of errors caused by search engine bot request to this endpoint with html request.

What's the best practice for dealing with this? Should I just respond with error format or I should create a page just because that search engine bot would crawl? And I am using Rails, specific recommendations would be nicer.

Thanks a lot.

larryzhao
  • 3,173
  • 2
  • 40
  • 62
  • Ever heard of `robots.txt`? AFAIK, it typically resides in Rails project's `public` folder. – D-side Nov 08 '14 at 16:38
  • @D-side Yes, I know that, I am just curious whether I should tell the search engine bot 'You are wrong' or I should fulfill it, mainly SEO wise. – larryzhao Nov 09 '14 at 11:20

1 Answers1

1

The best practice here is to deal with them with your proxy server (e.g. nginx). What you can do here:

  • create robots.txt file within your public directory and create appropriate rules (see here). However, since they're just rules, they do not have to be obeyed by any of them.
  • create nginx rule to reject requests looking like bots using $http_user_agent, e.g: Blocking all bots except a few with Nginx
Community
  • 1
  • 1
blelump
  • 3,233
  • 1
  • 16
  • 20