The Golden Hat

If you’re not the patient type or if you have bosses demanding greater search visibility, then quickly and accurately indexing the various pages of your site is something you might be interested in. Traditionally it can take the major search bots several weeks or even months to make a pit stop at your site to crawl and index the content. For sites with lots of fresh content or dynamic products, such as e-retailers, getting your site indexed frequently is also important. Yahoo! Search Submit Pro is one way of feeding new pages and/or fresh content to search engines, but you also have to pay for it.

A free and effective way of notifying major search engines about new pages, or just giving them a road map to all of the URLs in your site for crawlability purposes, is by using tools such as Google Sitemaps. Not only can you let Google know that you have (x) amount of pages and give Google Bot a map for crawling and indexing, but you can give Google additional information about your site such as how frequently content is updated and which pages in your site are most important. All of this information helps Google Bot determine how often it should crawl your site, and which pages carry the most “juice” in terms of value to your site.

The process is pretty easy. (1) Create a list of URLs in your site (2) Generate an XML version of the list . If you don’t know how to create the XML document yourself then use Google’s software to do it for you (3) Upload the Sitemap to your servers and submit the server file location in Google’s Webmaster Tools section for XML Sitemaps. By giving Google the location of your Sitemap file Google Bot will then reference the file and crawl the pages indicated.

Make sure that in your Robots.txt file you also reference the Sitemap file location as a double-check so that future crawls by Google Bot will direct the spider to your Sitemap. The reference is very simple and can look like this in your Robots.txt file:

User-agent: *
Disallow: /scripts/
Disallow: /help/
Disallow: /OrderStatus/

Sitemap: http://www.domain.com/sitemap.xml

Lastly, if you’re a WordPress blogger you can find a great Sitemap generator in the Plugins section. This plugin automatically generates an XML Sitemap file of your blog pages that a user can then submit to Google Webmaster Tools.

Cheers!

Like this article? Share, save, and spread it!