Stack Overflow
Stack Overflow

Questions tagged [bingbot]

Bingbot is a web-crawling robot, deployed by Microsoft to supply Bing. It collects documents from the web to build a searchable index.

23 questions
0
votes
0 answers

Why is Bingbot and Google Bot looking for a Robots.txt folder?

Our robots.txt file contains only this: User-Agent: * Disallow: / However, we've been receiving the below errors frequently. Not sure why both Bingbot and Googlebot are looking for a Robots.txt folder IP: 207.46.13.42 HTTP status code: 404 URL:…
Michael Merrell
  • 1,006
  • 8
  • 21
0
votes
1 answer

my website does not get visited by google bots?

I am trying to understand why my website does not get visited my google bots. http://www.nateiss.com/ I used Site-Analyser to analyse my site - you can see the website…
EKK
  • 171
  • 2
  • 12
0
votes
1 answer

How to fight against bingbot/2.0

I am tired by bingbot, howto fight against this robot. it create too session. I do that in ht access : opinion ? #bingbot create too session RewriteCond %{HTTP_USER_AGENT} ^bingbot/.* [OR,NC] RewriteCond %{HTTP_USER_AGENT} bingbot…
Kurama
  • 37
  • 1
  • 16
0
votes
1 answer

ERROR TYPE: System.IO.PathTooLongException FROM IP ADDRESS: 157.55.39.175 AKA Microsoft Bingbot

The details of the error notification are below. My question is how can I curtail the overwhelming number of Exceptions generated by Bingbots? What do I stand to lose, if anything, by doing so? System.IO.PathTooLongException IP ADDRESS:…
Erik Tammaru
  • 40
  • 1
  • 6
0
votes
0 answers

Bingbot keeps trying to access robots file in wrong directory

I keep getting BingBot trying to access the file below. There is no robots.txt file at that location (http://test.com/8329/test/documents). How can I stop this? C:\Test\Website\8329\test\documentsrobots.txt Exception System.Web.HttpException: A…
Mike Flynn
  • 22,342
  • 54
  • 182
  • 341
0
votes
1 answer

Httpd Access Log shows strange bingbot

i have a server set up in the cloud (ec2) the hosts all of my WordPress sites. i noticed today that the site were getting hit by a hack.. 109.87.118.222 - - [16/Oct/2013:13:10:31 -0400] "POST /wp-login.php HTTP/1.0" 200 3954…
JD Vangsness
  • 697
  • 3
  • 10
  • 27
0
votes
1 answer

Googlebot and Bingbot crawling DNN site

I have a DNN site with over 20,000 pages. The Googlebot and Bingbot are consistently crawling my website. When I look at my sitelog I can see that google and bing are crawling my site via the pageid (ex: www.url.com/Default.aspx?TabID=5000) The…
Cesar
  • 139
  • 2
  • 15
0
votes
2 answers

Bingbot ignoring robots.txt and attempting to retrieve a trafficbasedsspsitemap.xml

I have an app whose content should not be publicly indexed. I've therefore disallowed access to all crawlers. robots.txt: # Robots shouldn't index a private app. User-agent: * Disallow: / However, Bing has been ignoring this and daily requests a…
Brad Koch
  • 19,267
  • 19
  • 110
  • 137
1
2