9

I have noticed in my logs that Chrome requested a robots.txt alongside everything I expected it to.

[...]
2017-09-17 15:22:35 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8080
2017-09-17 15:22:35 - (sanic)[INFO]: Starting worker [26704]
2017-09-17 15:22:39 - (network)[INFO][127.0.0.1:36312]: GET http://localhost:8080/  200 148
2017-09-17 15:22:39 - (sanic)[ERROR]: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sanic/app.py", line 493, in handle_request
    handler, args, kwargs, uri = self.router.get(request)
  File "/usr/local/lib/python3.5/dist-packages/sanic/router.py", line 307, in get
    return self._get(request.path, request.method, '')
  File "/usr/local/lib/python3.5/dist-packages/sanic/router.py", line 356, in _get
    raise NotFound('Requested URL {} not found'.format(url))
sanic.exceptions.NotFound: Requested URL /robots.txt not found

2017-09-17 15:22:39 - (network)[INFO][127.0.0.1:36316]: GET http://localhost:8080/robots.txt  404 42
[...]

I am running Chromium:

60.0.3112.113 (Developer Build) Built on Ubuntu, running on Ubuntu 16.04 (64-bit)

Why is this happening? Can someone elaborate?

zython
  • 1,176
  • 4
  • 22
  • 50
  • 1
    Note that it’s `robots.txt`, not `robot.txt`. – unor Sep 17 '17 at 22:37
  • Can you reproduce this or is this just something you saw in your logs? – Josh Lee Jan 15 '18 at 12:37
  • I've since switched browswers, but I could reproduce it by restarting the server on my local machine and sending a request. – zython Jan 15 '18 at 14:53
  • Do you have a meta tag something like: `` – Phil Jan 18 '18 at 16:40
  • Do you have any Chrome extension that could be doing the request for you? Have you checked on the Network tab of Chrome DevTools when the request occurs? – Daniel Jan 20 '18 at 23:49
  • 2
    @MondKin it was a while back, i havent considered this before (silly me). sure enough i checked today what extensions I could have used and identified wappalyzer as the culprit https://github.com/AliasIO/Wappalyzer/blob/538e843b0dbe9816ef52b14386452aa725b1c054/src/drivers/webextension/js/driver.js#L273 if you put this in the answer ill reward you the bounty – zython Jan 21 '18 at 18:44

2 Answers2

9

There is the possibility it was not your Website that was requesting the robots.txt file, but one of the Chrome extensions (like the Wappalizer you mentioned). This would explain why it only happened in Chrome.

To know for sure you could check the Network tab of Chrome's DevTools to see at which point the request is made, and if it comes from one of your scripts.

Daniel
  • 21,933
  • 14
  • 72
  • 101
0

For chrome, there exists a plugin (SeeRobots) that checks if a robots.txt defines rules for search engines etc. - probably you have installed this plugin?

https://chrome.google.com/webstore/detail/seerobots/hnljoiodjfgpnddiekagpbblnjedcnfp?hl=de

Christoph Bimminger
  • 1,006
  • 7
  • 25