Robots.txt and Sitemap.xml

Since working with Google and Microsoft to support a single format for submission with Sitemaps, we have continued to discuss further enhancements to make it easy for webmasters to get their content to all search engines quickly. All search crawlers recognize robots.txt, so it seemed like a good idea to use that mechanism to allow webmasters to share their Sitemaps.

robots.txt 

 

A robots.txt file is a text file in a simple format which gives information to web robots (such as search engine spiders) about which parts of your website they are and aren't allowed to visit.

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots.
The "Disallow: /" tells the robot that it should not visit any pages on the site. 

For Example: if your site is at http://www.testsite.com/ then the robot.txt must go at

http://www.testsite.com/robot.txt

Sitemap linking in robot.txt

This can be used to tell search engines or other robots where your sitemap is located.

For example the complete robots.txt could look like this: 

User-agent: *
Disallow:
SITEMAP: http://www.testsite.com/sitemap.xml

 

SiteMaps

Whereas robots.txt files are usually used to ask robots to avoid a particular part of your site, a sitemap is used to give the robot a list of pages that it is welcome to visit.

By giving the search engine a sitemap you can (hopefully) increase the number of pages that it indexes. As well as telling the search engine the URLs of your pages, the sitemap can also tell the robots when the page was last modified, the pages priority, and how often the page is likely to be updated.

We can simple use Sitemap Generator to generate the sitemap of our website.

Sitemaps for multiple domains

If you have multiple websites, you can simplify the process of creating and submitting Sitemaps by creating one or more Sitemaps that includes URLs for all your verified sites, and saving the Sitemap(s) to a single location. All sites must be verified in Webmaster Tools. You can choose to use:

• A single Sitemap that includes URLs for multiple websites, including sites from different domains. For example, the Sitemap located at http://host1.example.com/sitemap.xml can include URLs for the following sites:
  • http://host1.example.com
  • http://host2.example.com
  • http://host3.example.com
  • http://host1.example1.com
• Individual Sitemaps (one for each site) that all reside in a single location. For example:
  • http://host1.example.com/host1-example-sitemap.xml
  • http://host1.example.com/host2-example-sitemap.xml
  • http://host1.example.com/host3-example-sitemap.xml
  • http://host1.example.com/host1-example1-sitemap.xml