Belchfire Search Engine Marketing

Few web sites have one but a sitemap can be a valuable tool to help search engines properly index your site.

A sitemap is basically a single or multiple web page that contains links to your homepage and all the other pages on your site. While search engine spiders are generally quite efficient at finding and following links in a given web site, a sitemap gives them a map to go by and ensures that all the pages you want indexed will be indexed.

If you have a web site with only a few pages, a sitemap can be created manually. If you have hundreds or even thousands of web pages, a program or script that generates a sitemap is recommended.

For an example, see the sitemap page for Belchfire.net here. Notice that the page contains only links who’s names are based on the page titles. It’s generally a good idea to indent pages hierarchically, and to breakup the sitemap into multiple pages if you have a lot of pages.

Belchfire.net has around 220 000 web pages and unfortunately I haven’t found a sitemapping program that can accommodate this number of pages let alone break the sitemap up into smaller pieces (so my own sitemap doesn’t adhear to this rule). You don’t want more than perhaps 1000 links on a single page as some spiders may stop following links past a certain point.

Here are some free and commercial sitemapping tools to try:

• Netroglycerine Sitemap Tool - free - what I used to generate the Belchfire.net sitemap - script died after around 3700 links
• Xenu’s Link Sleuth - free - more of a link checker but can optionally generate a sitemap during the final report - froze up trying to run it on Belchfire.net after around 65 000 links checked
• SiteTree - free - command line java program and classes that can generate a sitemap by traversing a directory on a hard drive (looks good, but won’t be of much use if you don’t have command line access to your web server - not sure about the accuracy either)
• Extreeme SiteXpert - commercial - seems to be the best commercial offering with dozens of options, templates and formats - demo limited to 15 pages
• Site Map Pro - commercial - another good commercial offering - demo limited to 10 pages

You can even get fancy with some of these commercial offerings by having them add the first few lines of text under each link. Search engines will love the content and your sitemap may even get higher page rankings than some of your content pages. Really, a site map may or may not be of any real benefit to your human visitors (assuming you already have a decent navigation stucture), but adding a sitemap can help improve the number of pages indexed by search engines by providing them with a starting point and a map of files to index. Don’t forget to add an appropriate title and meta tag info on your sitemap page(s)!

The proof’s in the pudding…

Since adding a sitemap to Belchfire.net, indexed pages on the major search engines has increased from 25 000 to over 52 000 (Alltheweb, AltaVista, Google, Yahoo, Inktomi combined) and my web site traffic has increased by over 30% (an additional 1500 unique visitors per day).