Every Portal site wants to be "known" and "popular" in the search engines.
AOL, BING, GOOGLE are the big ones .. but there are dozens of other search engines in the game.

Search engines work by "crawling" web site content (pages, text, hyperlinks and images).

This "crawling" is performed by back-end programs that run on servers and perform the crawling actions.

Crawling means the back-end program (BOT) is acting very much like an HTTP Web Browser. However, the BOT is not a human who is viewing the result of the web page. The BOT is a program that reads and processes the underlying HTML web page scripts and EXTRACTS meaningful CONTENT.

Once the target content is extracted from a web page, it is INDEXED (parsed, processed, categorized, sorted, rated) and then STORED in a large database.

Each Search Engine has its own method (algorithm) for rating and scoring web pages based on several variables or factors. This include the popularity of the web page (how many times that web page is searched and selected within the search results), the frequency of the web page in terms of cross-linked pages that reference the page, and the overall popularity of the keywords that are found within the page content.

When visitors search for keywords in a search engine, the keywords are found within the INDEX, and the resultant web site page URL links are presented in the search results with the most popular links on top.

The best way to get a high rating within the search engines is to have fresh and interesting content that is referenced and inter-linked by dozens (hundreds) of other web sites. In this way, the more popular your web pages are, the more popular they will become.

The growth rate of "popular" web sites is, in fact, exponential.

Here, in reverse order of how likely they are to visit any web site, are the 10 most important good bots that you should know about now. Make sure your security strategy welcomes these bots (or at least know why you chose to block them)!