The search engines work as a gigantic indexing system for the World Wide Web. Just as a database in a library indexes and stores all its books, the search engines store most of the world’s websites. Each search engine run automated programs often called “bots” or “spiders” that use the hyperlink between web pages to “crawl” and then index the pages within their massive database. It’s estimated that there are around 20 billion web pages in the World Wide Web, and that the search engines have crawled around 8-10 billion of them.
When someone searches for a keyword/phrase the search engines process the queries, match them to the most appropriate results, according to the search engine algorithm (the mathematical equation of how to rank a site). And lastly sort and display the results according to relevance, from most relevant at the top to the least at the bottom.
In the below illustration (which is genius by the way) you have two search engine spiders (one Google spider and one Yahoo spider, how cute!) following the links between tube stations (websites), the tube line is the link between station to station (website to website).
Illustration courtesy of SEOmoz, (thanks Rand, I love this illustration!)
Crawl then Index
When a search engine spider visit a website it reads all the content and follows all the links internally, and then it decides what to index the webpage for. This is why it is SO crucial to have search engine optimised content on your website. If there is no search engine visible content on your website (mostly meaning content in HTML), the search engines won’t know what to index your website for and you might be indexed for keywords that are not related to your business.
Recommended Tools
To see how the search engines view your website check out SE Spider.com
For finding out whether the search engines have any problems crawling your site check out SEOmoz’s Crawl Test Tool
Any tools you want to share with us? Or any SEO related questions about your website please leave a comment!
Next week: learn about the difference between being indexed by the search engine and ranked highly in the SERPs (Search Engine Results Pages)!
When someone searches for a keyword/phrase the search engines process the queries, match them to the most appropriate results, according to the search engine algorithm (the mathematical equation of how to rank a site). And lastly sort and display the results according to relevance, from most relevant at the top to the least at the bottom.
In the below illustration (which is genius by the way) you have two search engine spiders (one Google spider and one Yahoo spider, how cute!) following the links between tube stations (websites), the tube line is the link between station to station (website to website).
Illustration courtesy of SEOmoz, (thanks Rand, I love this illustration!)
Crawl then Index
When a search engine spider visit a website it reads all the content and follows all the links internally, and then it decides what to index the webpage for. This is why it is SO crucial to have search engine optimised content on your website. If there is no search engine visible content on your website (mostly meaning content in HTML), the search engines won’t know what to index your website for and you might be indexed for keywords that are not related to your business.
Recommended Tools
To see how the search engines view your website check out SE Spider.com
For finding out whether the search engines have any problems crawling your site check out SEOmoz’s Crawl Test Tool
Any tools you want to share with us? Or any SEO related questions about your website please leave a comment!
Next week: learn about the difference between being indexed by the search engine and ranked highly in the SERPs (Search Engine Results Pages)!
0 TrackBacks
Listed below are links to blogs that reference this entry: How do Search Engines work?.
TrackBack URL for this entry: http://www.b2bm.biz/blogadmin/mt-tb.cgi/15
Leave a comment