0

I have an app that I need search engine crawlers to be able to index. I don't need that the whole app to be indexed just specific URL (or URL pattern) like http://examplegwtapp.com/xyz where xyz is a hash code, similar to those URL shorteners out there.

My app works like this:

When that URL is accessed, the servlet will forward the request to a GWT app passing this fragment: app.html#View?hash=xyz

So the View page is totally dynamic.

The question is what is the correct way to make this specific dynamically generated URL to be indexed or crawled by search engines?

quarks
  • 33,478
  • 73
  • 290
  • 513

1 Answers1

1

I would look into Making AJAX Applications Crawlable and A proposal for making AJAX crawlable

In a nutshell these are the steps you should consider:

  1. Crawlers maps from pretty url to ugly url. i.e from http://examplegwtapp.com/app.html#View?hash=xyz to http://examplegwtapp.com/app.html?_escaped_fragment_=hash=xyz
  2. crawler requests the ugly url
  3. Server maps from ugly url to pretty url. you can do this by identifying on the apache level any _escaped_fragment_ request and redirect it to a certain server controller to handle the crawler call. i.e. check Apache rewrite condition for ajax crawling
  4. Server invokes headless browser(HtmlUnit or if just small portion of your code is js just generate the static html with your server code. i.e HtmlUnit Generate Page for GWT App
  5. Headless browser response returned to the crawler.
Community
  • 1
  • 1
liorsolomon
  • 1,785
  • 14
  • 24