Questions tagged [sitemap]

A sitemap is an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. The sitemap protocol format is defined at http://www.sitemaps.org/protocol.html

Sitemap can be either a document in any form used as a planning tool for web design, or a web page that lists the pages on a web site, typically organized in hierarchical fashion. This helps visitors and search engine bots find pages on the site.

While some developers argue that site index is a more appropriately used term to relay page function, web visitors are used to seeing each term and generally associate both as one and the same. However, a site index is often used to mean an A-Z index that provides access to particular content, while a site map provides a general top-down view of the overall site contents.

XML is a document structure and encoding standard used, amongst many other things, as the standard for webcrawlers to find and parse sitemaps. There is an example of an XML sitemap below (missing link to site). The instructions to the sitemap are given to the crawler bot by a Robots Text file, an example of this is also given below. Site maps can improve search engine optimization of a site by making sure that all the pages can be found. This is especially important if a site uses a dynamic access to content such as Adobe Flash or JavaScript menus that do not include HTML links.

They also act as a navigation aid by providing an overview of a site's content at a single glance.

Source: https://en.wikipedia.org/wiki/Site_map

2093 questions
16
votes
3 answers

Should I list PDFs in my sitemap file?

Should I add PDFs to my XML sitemap? I want to know if Google will crawl the PDFs.
user132114
15
votes
11 answers

How to dynamically create a sitemap.xml in .NET core 2?

Can anyone tell me how to create a Sitemap in .NET Core 2? These articles/alternate link are not working in .NET Core 2.
Beginner
  • 189
  • 1
  • 1
  • 7
15
votes
4 answers

'TemplateDoesNotExist' error with creating Sitemap for Django app

I followed the sitemap activation steps on the django site but i keep getting a 'TemplateDoesNotExist' error. Maybe i am misunderstanding, but isn't genericview supposed to generate the page? ########### url.py…
ironic_username
  • 205
  • 3
  • 9
14
votes
1 answer

Does the url order matter in a XML sitemap?

For search engines and website crawlers, does the url order matter in a XML sitemap? Currently when the sitemap is generated, I order the website urls sequentially using a unique id, in the database. Should I order the urls in date…
stukelly
  • 4,257
  • 3
  • 37
  • 44
13
votes
4 answers

Confusion about sitemap_index.xml

I've read the Google docs on sitemap formats but there's one thing they don't make clear: will search engines automatically look for and find /sitemap_index.xml or do you have to tell them about it via /robots.txt or the main /sitemap.xml? Can you…
Teflon Ted
  • 8,696
  • 19
  • 64
  • 78
12
votes
6 answers

how to generate a sitemap in expressjs

I downloaded my XML sitemap from the sitemap xml generator website. I placed my sitemap.xml on my public directory but when I tried to submit the sitemap.xml into google console i received the following error: General HTTP error: 404 not found HTTP…
Vic B-A
  • 338
  • 1
  • 3
  • 10
12
votes
6 answers

Generating a Visual Site Map of an Existing Site

I thought I could easily answer this question by searching the web, but my Google-fu has failed me. So I turn to you, oh mighty stackoverflowers, and beseech you for enlightenment: What is the quickest and best way to generate a visual site map…
user18359
  • 4,673
  • 5
  • 19
  • 11
11
votes
1 answer

How to return a .xml file in a Laravel Route

I have generated a site map .xml file and want to return. I placed the file in my views folder and made this route: Route::get('/site-map', function(){ return view('sitemap.xml'); }); Although I just cant get it to display.
Victori
  • 339
  • 2
  • 5
  • 17
11
votes
3 answers

Sitemap randomly breaks over time

I've been having some production runtime errors that I don't fully understand. This has happened to us on a couple different ASP.NET 4.0 Web Sites (shudders - yes, I know - we're porting it to MVC but that's taking some time). First of all, we have…
Jaxidian
  • 13,081
  • 8
  • 83
  • 125
11
votes
3 answers

Visualizing the SiteMap of a large (page number) website

I'm looking for a tool or service that can spider a web domain with a large number of pages, create a sitemap, and then visualize that map in a way that will help me see, understand and group content (I'm new to the site) Something like a tree-view…
Michael
  • 1,133
  • 3
  • 15
  • 30
11
votes
4 answers

Why use a Google Sitemap?

I've played around with Google Sitemaps on a couple sites. The lastmod, changefreq, and priority parameters are pretty cool in theory. But in practice I haven't seen these parameters affect much. And most of my sites don't have a Google Sitemap…
Steve Wortham
  • 21,740
  • 5
  • 68
  • 90
11
votes
5 answers

How to make a sitemap link in the page head pass the W3C validator?

I'm trying to pass a page through the W3C Validator. The validation fails on the sitemap, which I'm including like this: The error I'm getting is: Bad value…
frequent
  • 27,643
  • 59
  • 181
  • 333
10
votes
1 answer

Proxy a file from S3 with Heroku and Rails

I need to send a file stored in S3 through my rails application hosted on heroku. I cannot use a redirect to the URL of the file on S3 as I want to send a sitemap which should be served from the same domain as the links inside. Ideally, I would use…
Adrien Coquio
  • 4,870
  • 2
  • 24
  • 37
10
votes
2 answers

Listing both sitemaps and sitemap index files in robots.txt?

My site is comprised of 3 main sections: Reviews, Forum, and Blog. I have plugins for the forum and blog that automatically generate sitemaps for them. The forum plugin generates a sitemap INDEX file pointing to multiple indexes, and the blog plugin…
Chris
  • 1,273
  • 5
  • 19
  • 33
10
votes
3 answers

Sitemap for a site with a large number of dynamic subdomains

I'm running a site which allows users to create subdomains. I'd like to submit these user subdomains to search engines via sitemaps. However, according to the sitemaps protocol (and Google Webmaster Tools), a single sitemap can include URLs from a…
bartekb
  • 203
  • 6
  • 14