3

I am using SEMrush for SEO purposes. I am unsure why I am receiving the following error on their site for crawling my page:

We couldn't crawl this page using the SEMrushBot user agent due to an HTTP error 406. Nevertheless, we were able to collect a few general ideas for this page. Please ensure that your page can be accessed by search engine crawlers, and then start optimizing it using our ideas.

What could be causing a 406 error to prevent bots from crawling the page? I believe that Google is able to crawl it because I don't have any errors on their side.

What can I check?

Robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /admin/
Disallow: /old-site/
sitemap: https://example.com/sitemap.xml
Paul
  • 3,348
  • 5
  • 32
  • 76

1 Answers1

0

406 is error which is telling you that the server won't give you any data becouse your user agent not in it accepted list.

The target resource does not have a current representation that would be acceptable to the user agent, according to the proactive negotiation header fields received in the request, and the server is unwilling to supply a default representation.

So basically the way to solve this is to change your user agent.

Volod
  • 1,283
  • 2
  • 15
  • 34
  • How could I add them to my accepted user agent list? – Paul Jun 06 '19 at 15:23
  • After answering I read about SemRushBot and it's seems to me that you can't change it setting because you have no access to settings or source code, also you probably can't change settings on the site of a site that you want to crawl. So for now it seems impossible to me. Sorry for disappointing you. Maybe some workaround exist or someone with more experience in this field will give you more optimistic answer. – Volod Jun 06 '19 at 15:29
  • Read your question one more time. If it's your site check your robots.txt file and add it to body of your question please – Volod Jun 06 '19 at 15:30
  • I added the robots.txt file to the question. – Paul Jun 06 '19 at 15:43
  • You receive 406 both for http and https? Which server do you use? – Volod Jun 06 '19 at 15:58
  • My site only renders in https. That is how I have it setup in the htaccess file. I am on a Linus server. It is hosted by hostgator. A different site I have (on hostgator as well) has no issues with this 406 error. – Paul Jun 06 '19 at 16:05