robots.txt and sitemap files defined

One of the more confusing aspects of Search Engine Optimization (SEO) is dealing with the files that control the search engine spiders or crawlers. This article will show you the uses and basic structure of each file.

A robots.txt file tells search engines what you do not want them to index. It can include commands that pertain to all search engines, or targeted to specific crawlers.

Example:

User-agent: *
Disallow: /cgi-bin
Disallow: /eklog
Disallow: /images
User-Agent: MSNbot
Crawl-Delay: 10

The first 4 lines are directed to all crawlers. The last 2 lines tell MSNbot to delay indexing 10 seconds between pages to prevent overloading your site.

In contrast to a robots.txt file, a sitemap file tells the search engine crawlers what you do want them to index. The search engines are very picky about the format, especially if you use the XML type. That one lists the file name, date of last change, revisit schedule and the importance on a scale of 0.1 to1.0

Example:

<?xml version=”1.0″ encoding=”UTF-8″?>
<urlset xmlns=”http://www.google.com/schemas/sitemap/0.84″>
<url>
<loc>http://www.websitemanagers.net/</loc>
<lastmod>2008-05-01</lastmod>
<changefreq>monthly</changefreq>
<priority>1.0</priority>
</url>
</urlset>

There are online tools to help you generate that file. Google and Yahoo have them, plus a plugin for WordPress is available free that will automate the process on your blog.

A much simpler version is a plain sitemap.txt file that lists just the URLs within your site.
Example:

http://www.websitemanagers.net/
http://www.websitemanagers.net/bloghosting/
http://www.websitemanagers.net/color-hex-codes.php
http://www.websitemanagers.net/sitemap.php

For more information on creating and managing robots.txt files, visit Wikiedia. Sitemap instructions can be found at Google Webmaster Tools.

Note that both the sitemap and robots.txt file require using either a file manager, such as the one built into cPanel, or a text editor combined with an FTP (file transfer) program.

Jim Hutchinson
Website Managers - Discount Website/Blog Hosting with cPanel