Stack Overflow
Stack Overflow

Questions tagged [search-engine-bots]

51 questions
0
votes
0 answers

Crawling the dynamic web application by bots

I have a simple public facing web application and I need to make the my site searchable by search bots to scan. So I plan to modify my application by adding meta tags, titles and other descriptions tags in my web pages so that search engines can…
KV MC
  • 29
  • 4
0
votes
1 answer

How search Engines uniquely identify each pages on web

If I write a post who's link is 'example.com/abc.php' & this post is ranking in Search Engine. After some days I update this post including its content & also its permalink to 'xyz.php'. So now the post which had URL 'example.com/abc.php' before is…
Abhishek Kamble
  • 1
  • 2
0
votes
2 answers

ASP.Net how to allow searchbots(google) to skip a disclaimer page

I am creating an ASP.NET website witch forces users to accept a disclaimer. When they accept the disclaimer a cookie and session is set. On each page request a check is fired to see if the session or cookie present. We want to allowed Google and…
Ivo
  • 3,406
  • 4
  • 33
  • 56
0
votes
0 answers

Is JSON status request AJAX SEO friendly

I implemented a feature on a site that in short is the following: There should be a pop-up dialog after page is visited more than 3 times. The pop-up is a simple dialog with one button which is actually anchor(link). All the content(HTML Javascrpt,…
Hairi
  • 3,318
  • 2
  • 29
  • 68
0
votes
1 answer

Extracting Results from "Result-Page" With Selenium in Python

Unfortunately I have a little problem with the realization of my Python program. At one point I can't get any further. The program should do the following: Perform an automatic search for a specific keyword on the search engine…
user11914084
0
votes
1 answer

Google not indexing a geo restricted page distributed by CloudFront

I have a website hosted on AWS using CloudFront that, due to some legal limitations, needs to be only accessible in the UK and Ireland. After setting up geo restriction in CloudFront and submitting the domain to Google via the Webmaster Tools (at…
Adam P
  • 139
  • 1
  • 10
0
votes
1 answer

how to stop search engines linking to HTTPS?

Background I manage several websites which have no SSL certificate, hosted on a LAMP shared environment, alongside some that do. The shared environment has an SSL certificate for all sites, but it's an invalid domain unless connected to with a…
totallyNotLizards
  • 8,489
  • 9
  • 51
  • 85
0
votes
1 answer

Disallow query strings in robots.txt for only one url

so I have one url, chickens.com/hatching that has potential query strings it could be indexed with, i.e. chickens.com/hatching?type=fast. I would definitely like to keep the base url, chickens.com/hatching indexed, but no query parameters. I would…
ambe5960
  • 1,870
  • 2
  • 19
  • 47
0
votes
1 answer

Search bots creating issue in web service call through curl

I implemented curl code to fetch to live inventory from the vendor web service when any user opens the product detail page. But search engine bots are accessing the page and this results in curl call to web service 1000 times every second. How to…
Inder Singh
  • 51
  • 3
0
votes
1 answer

Block certain Addresses from being Crawled Robots.txt

I've been asked (in order to improve SEO results) to remove a certain web address from a Robots.txt. I'm struggling (SEO is not my forte) to figure out how to block a certain domain, even though both addresses point to the same website. For…
Narratones
  • 35
  • 4
0
votes
1 answer

Doesn't google execute javascript if I add in html?

I have built site with angularjs and I am using html5 pushstate mode. I have also added tag in html page. I don't provide html snapshots for bots. I am expecting google to execute javascript itself and index my site. But, google is indexing the…
Sachin Nanaware
  • 13
  • 5
0
votes
1 answer

Ajaxsnapshot's proxy interfering with CloudFlare's proxy

We are using CloudFlare service for CDN, Security and other services. And we are using Ajaxsnapshot for creating snapshots for Search Bots. The problem is we are getting Error 1000 - DNS points to incorrect IP. When we switch off CLoudFlare…
Swati Chopra
  • 21
  • 2
0
votes
1 answer

I want to redirect 15000 pages of my website

I have a social networking website, and I want to redirect about 15000 previous web pages. I have been suggested by other experts to use 301 redirect. All I want to know is that how redirection will not effect search engine ranking, is there any…
zohair sharif
  • 9
  • 1
0
votes
1 answer

IIS set up effecting bots which is impacting on search results

This is a tricky one to explain. I believe the google bot is getting confused because of the way iis/sites are set up. The actual issue is, when searching Google and the result is www.someSiteURL.com the description underneath is: A description…
Mat41
  • 1,287
  • 4
  • 15
  • 29
0
votes
1 answer

Webmaster Tools doesn't like my page set pattern

I'm trying to use the highlighting feature on Webmaster tools. I got done filling it out for my page, but when I go and try to create the page set, it doesn't find any files matching the pattern. The default pattern that google chose…
ganders
  • 7,285
  • 17
  • 66
  • 114
1 2
3
4