Questions tagged [sitemap]

A sitemap is an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. The sitemap protocol format is defined at http://www.sitemaps.org/protocol.html

Sitemap can be either a document in any form used as a planning tool for web design, or a web page that lists the pages on a web site, typically organized in hierarchical fashion. This helps visitors and search engine bots find pages on the site.

While some developers argue that site index is a more appropriately used term to relay page function, web visitors are used to seeing each term and generally associate both as one and the same. However, a site index is often used to mean an A-Z index that provides access to particular content, while a site map provides a general top-down view of the overall site contents.

XML is a document structure and encoding standard used, amongst many other things, as the standard for webcrawlers to find and parse sitemaps. There is an example of an XML sitemap below (missing link to site). The instructions to the sitemap are given to the crawler bot by a Robots Text file, an example of this is also given below. Site maps can improve search engine optimization of a site by making sure that all the pages can be found. This is especially important if a site uses a dynamic access to content such as Adobe Flash or JavaScript menus that do not include HTML links.

They also act as a navigation aid by providing an overview of a site's content at a single glance.

Source: https://en.wikipedia.org/wiki/Site_map

2093 questions
24
votes
3 answers

Serving sitemap.xml and robots.txt with Spring MVC

What is the best way to server sitemap.xml and robots.txt with Spring MVC? I want server these files through Controller in cleanest way.
michal.kreuzman
  • 12,170
  • 10
  • 58
  • 70
21
votes
4 answers

Sitemap generation with Codeigniter

I need to generate a sitemap in a Codeigniter application. I found a few libraries but all of them are outdated and have bug. Do I really need a separate library for this? I want to know the best way to generate the sitemap in Codeigniter.
bmrinal
  • 365
  • 3
  • 4
  • 10
20
votes
7 answers

Google search console fails to fetch sitemaps | "Sitemap could not be read"

I have generated a sitemap from online generators, it seems to be working and even i tested it on old google search console sitemap testor and it works. but when i submit it in both versions it just displays error message.
user9480491
20
votes
2 answers

Django Sitemaps and "normal" views

Maybe I didn't understand the purpose of Sitemaps or maybe I didn't understand how to use sitemaps. Right now my sitemap is including all "dynamically" created pages, like the blog posts. But how do I add "static" pages like my index and contact…
Clash
  • 4,896
  • 11
  • 47
  • 67
19
votes
2 answers

Manually add sitemap located in s3 into google webmaster tools

I have an app running in Heroku. I am using sitemap_generator to generate sitemap and save it into s3. I have added the robots.txt to contain my sitemap location. My question are. How can I know my sitemap are successfully find by search engine…
stanicmail
  • 743
  • 7
  • 19
19
votes
1 answer

How to let sitemap generator fully crawl Angular router for SPA?

I am trying to generate a sitemap for my webpage. The sitemap generators online only show me a homepage on xml file. -
GreedyAi
  • 2,709
  • 4
  • 29
  • 55
19
votes
1 answer

namespace prefix xhtml on link is not defined - sitemap.xml

I am trying to add language to my sitemap.xml file, but I get a "namespace prefix xhtml on link is not defined" error. How do I defined it? I cannot find anything useful on google. Here is the file:
crystyxn
  • 1,411
  • 4
  • 27
  • 59
18
votes
4 answers

How to efficiently serve massive sitemaps in django

I have a site with about 150K pages in its sitemap. I'm using the sitemap index generator to make the sitemaps, but really, I need a way of caching it, because building the 150 sitemaps of 1,000 links each is brutal on my server.[1] I COULD cache…
mlissner
  • 17,359
  • 18
  • 106
  • 169
17
votes
3 answers

Should I include paginated results in my sitemap.xml?

I have listing pages that take a page argument on the url like the following: http://www.domain.com/foo/bar/?page=7 Should I just include the URL without params or should I list all pages in my sitemap.xml? EDIT Paginated content are listings, like…
muhuk
  • 15,777
  • 9
  • 59
  • 98
17
votes
2 answers

How to create a sitemap using PHP & MySQL

I was wondering how can I create a sitemap using PHP & MySQL and is there any sitemap design examples you know of?
HELP
  • 14,237
  • 22
  • 66
  • 100
17
votes
3 answers

MVC: How to route /sitemap.xml to an ActionResult?

I've got a SitemapActionResult that overrides the ActionResult, and delivers a SEO sitemap.xml when http://www.sprelle.no/Home/SiteMap is hit. So far so good. What I would like, though, is to serve the sitemap.xml when Google visits /sitemap.xml.…
HaakonL
  • 173
  • 1
  • 1
  • 4
16
votes
4 answers

How do you notify Google through code that there is a updated sitemap?

On this page of Google help: https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html#submitting Google mentions that there is a way to notify them of an updated sitemap using an HTTP request. When you click the link, it takes you to…
Darryl Hein
  • 142,451
  • 95
  • 218
  • 261
16
votes
1 answer

Sitemap and object with multiple urls

The normal way sitemap is used in Django is: from django.contrib.sitemaps import Sitemap from schools.models import School class SchoolSitemap(Sitemap): changefreq = "weekly" priority = 0.6 def items(self): return…
Alexander Tyapkov
  • 4,837
  • 5
  • 39
  • 65
16
votes
6 answers

How to scrape all contents from infinite scroll website? scrapy

I'm using scrapy. The website i'm using has infinite scroll. the website has loads of posts but i only scraped 13. How to scrape the rest of the posts? here's my code: class exampleSpider(scrapy.Spider): name = "example" #from_date =…
Michimcchicken
  • 543
  • 1
  • 6
  • 21
16
votes
3 answers

ASP.NET web.sitemap - roles do not seem to control visibility?

I have a (simple) sitemap and am trying to figure out why a certain sub-menu is always visible.
Keith Barrows
  • 24,802
  • 26
  • 88
  • 134