0

I have a few files that can be downloaded from my website. Every few hours all files are downloaded from the same IP address (98.137.207.97 - might be Yahoo?). How should I proceed? Should I just block the IP (and hope the downloads are not taken up by another IP) or are there better ways to handle this?

IronGoofy
  • 201
  • 1
  • 2
  • 7

1 Answers1

1

You should create a robot.txt file if you don't want bots/spiders/etc to be crawling certain parts of your site.

Jason
  • 718
  • 5
  • 15