Google offers the ability for webmasters to submit guideline information to assist its robots in indexing websites. You can even instruct the Google robot to prioritise certain pages. This is called the Google Sitemap and I highly recommend taking advantage of this facility to ensure that Google is indexing all of your web pages.
If you use Drupal, then this task is a whole lot easier. There are a few things that you need to be aware of, so I have put together the following set of steps to help you along the way. If you do not use Drupal, then this guide will still be of use, but you will need to create your own sitemap.
1) Download the Google Sitemap (now known as XML sitemap) module from http://drupal.org/project/gsitemap
2) FTP the Google Sitemap to your module folder on the web server
3) Enable the Google Sitemap in Administer › Site building
4) Go to Administer › Site configuration > Google Sitemap
5) Create an account at http://www.google.com/webmasters/. Alternatively, you can use your Google Account log in if you have one
6) Go to Webmaster Tools
7) In the “Add a site” box, enter the URL of your site and hit OK
8) Click on “Verify your site” and chose the “Upload an HTML file” option. Google will give you an HTML file
9) Go back to your website and go to Administer › Site configuration > Google Sitemap
10) Under “Other Settings”, enter the HTML file name that Google provided in the verification link box
12) Also under “Other Settings”, ensure that the “Submit sitemap to Google when updated”, “Submit sitemap to Google on cron run” is NOT ticked. This is to prevent Google from thinking that you are spamming it
13) Copy the URL of the sitemap. This is likely to be http://www.yourdomain.com/gsitemap
14) Go to “URL aliases”. Create an alias for gsitemap. Call it gsitemap.xml. This is needed to conform with Google rules. Google requires the sitemap to be in the root folder. The URL http://www.yourdomain.com/gsitemap appears as if the sitemap is in the folder “gsitemap” and therefore does not comply with the rules. Creating an alias of gsitemap.xml ensures that the file appears as if it is in the root folder.
15) Go back to Google Webmaster Tools. Click on “Sitemaps” and then “Add a sitemap”
16) Choose “Add General Sitemap” and then your sitemap URL. For example: http://www.yourdomain.com/gsitemap.xml
That’s it! Google will verify the sitemap and use the information contained in it when its robots crawl your site. Eventually you will see how many URL’s have been submitted using the sitemap. There are a few other cool features in the Google Webmaster Tools that I will discuss in the near future.