Understanding Search Engine Crawlers

July 20, 2009 - PRLog -- Understanding the way a search engine crawler indexes pages and the specific algorithm elements involved in their programming is key to determining which optimization techniques to use. The algorithms use a combination of page content and structure, loading time, and analysis of inbound links to determine the page rank for keywords and phrases. To achieve the best results with search engines all aspects of the algorithm are needed.

There are many methods for controlling the way search engines will crawl and index your pages. I like to think of search engine crawlers reading a webpage like a human would read a book. First to find out what the book is about, a person would view the page title and summary (page description) of the book. Next a person would examine the chapters (page topics and titles) and next the headlines and then the paragraph content. It is important to build your site in a way so the crawlers understand the most important to least important information.

Some ways to control crawlers:

•.htaccess - Use to set up allowed Server Side Includes and 301 Redirects.
•sitemap.xml - Set up and verify with Google Webmaster Tools. Its fast, simple, and easy to schedule crawler visits and page importance. These tools also offer a way to analyze how the crawler sees a website.
•robot.txt - Set up to allow and disallow access to certain pages.
•Set up easy to follow or nofollow links - Search engine robots use links to crawl the world wide web. Both inbound and outbound links should be properly strucutured.

# # #

Web design, development, online marketing and web analytics services including search engine optimization and social media marketing