The Deal with Site Maps

For SEO and usability purposes, it is beneficial for your website to have two kinds of Site Maps. One site map is specifically for the benefit of users on your site, to help them navigate the different pages, and also to help organize and link all pages on your site together.

Ideally, this site map would be in HTML format, divided into sections, and provide clear links with descriptive text to each of the pages or sections of your site.

The second type of site map to upload to your site is specifically used by search engine spiders, to help locate and index the content of your site. This site map is in XML format, and is ideally uploaded to the highest directory possible on your site's server.

Google provides more information on site maps here: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40318

In a previous post I mentioned a helpful XML site map generator online at XMLsitemaps.com, but I have found that this tool limits you to 500 pages if you don't want the paid upgrade. In search of a more useful site map generating tool, I discovered the Audit My PC Free Google Sitemap Generator.

For free, this tool will thoroughly analyze the pages of your site and generate a comprehensive site map following rules you set up if you so choose (rules including "ignore images," "respect no-follow links," "respect links disallowed in robots.txt," etc.), and will allow you to export both the site map XML file itself, as well as a report on all site links. This will allow you to identify broken links and URLs on your site, as well as pages that time-out when loading. I highly recommend this tool for generating and analyzing your own sitemap.xml.

By the way, once you've created and uploaded your XML sitemap file, don't forget to make full use of it by uploading it to Google Webmaster Tools and Yahoo! Site Explorer. ;)

Using Google's Webmaster Tools

Google offers free webmaster tools to help you check for technical issues that may be barring your website from high natural rankings. Here's the skinny on what's available in Google's Webmaster Tools, and what you can do with them.

Note: To use all the webmaster tools available, you must have FTP/back-end access to your website. This is so you can verify your site on Google and utilize automatic diagnostic features that help you improve the indexing of your site by search robots.

1. Log into Google's Webmaster Tools with your Google Account info.

2. Type in the URL of the site you want to optimize in the Dashboard.

3. You will now see an Overview page, where you can follow instructions to verify your site by uploading a file to your server via FTP, or adding a meta-tag to the site's HTML. Verification will unlock diagnostics tools, statistics on your site, link information, and various other tools.

4. Now it's time to upload a sitemap in xml format to Google. It's easy to create your own site map online, or learn how to make one yourself using Sitemap Protocol. Once your sitemap.xml file has been created, upload it via FTP to the highest directory on your site's server. Now go to the Sitemaps section of WM Tools and click on the link to "Add a Sitemap."

5. After submitting your sitemap, go to Diagnostics > Web Crawl. This is where information is listed about errors or issues impeding Google's searchbot from crawling any page on your site. This includes any broken links, pages that time out before loading, and any restricted URLs. Using this tool to identify erroneous pages on your site will allow to pinpoint the problem to go in and repair site content.

6. I will skip Mobile Crawl as that is for websites designed for mobile devices - and since this is a fairly new invention, I'm sure this is not applicable to most of you.

7. Now go to Diagnostics > Content Analysis. This page deals with issues to do with the actual content on your site, including meta-tags and title tags. You can drill down and identify pages on your site with duplicate, long, or short meta-descriptions, or missing title tags, and any non-indexable content.

8. I'm honestly not so convinced of the value offered by the Webmaster Tools' "Statistics" section, but go ahead and poke around if you are interested in general Page Ranking on your site, what search queries are leading to your site on Google, and general info about subscribers (if you have any - though I would recommend FeedBurner for gathering those stats if you're serious about subscribers), or a list of indexed pages.

9. Ever wonder what pages on your site are being linked to externally? Check out the Links section of Webmaster Tools. You can also find out what Sitelinks appear for your site on Google's search results pages. For more information on how to get sitelinks up for your site, or what the heck sitelinks are, you can read some theories on how to get sitelinks to appear.

10. Upload a robots.txt file to your site's server if you want to prohibit Googlebots from crawling specific pages of your site. For example, block the bots from certain secure pages. Don't know what a robots.txt file is? Learn more about them here. Remember also to include a reference to the location of your sitemap in the robots.txt file.

11. You can generate and analyze your robots.txt file in the WM Tools "Tools" section.

And that about wraps up Webmaster Tools.