Crawling the Best Pages on the Web

Today, we’d like to share some thoughts on one of the major components of any search engine, and something we’ve been focusing on a lot lately, namely the Crawler.

How does a search engine find pages to include in the database? By automatically mass-surfing the web, jumping from page to page and grabbing the information found in the process. This is done by a program called a Crawler, Bot or Spider. Entireweb’s crawler, affectionately called Speedy Spider, simultaneously runs on a large number of machines and visits tens of thousands of pages at the same time, all day, every day.

"Garbage in, Garbage out" is an old engineering proverb: To build a great search engine, you need to start at the source, namely the way you select pages to include, at what schedule you visit them, and what you do with the stuff you find on the pages (that’s what’s often called Indexing). This is a problem we’re taking very seriously.

The ideal crawler makes sure to grab every page on the net that is of any interest to human searchers, and does so as soon as it has been created or updated. Our goal is to approximate this as closely as we can.

We feel there is no point in including pages that don’t carry any tangible information. For example, many search engines will include pages that basically contains nothing more than advertisements, simply because the search engine may benefit financially from showing such ads. These pages, which include things like parked domains, scraper sites and various kinds of web-spam, do little more than dilute the search results. We’ve also made innovations to duplicate content removal, so that the information you find in our search results will be truly unique, diverse and useful.