Skip to main content

Become.com's Web Crawler: A Massively Scaled Java Technology Application

No replies
kohsuke
Offline
Joined: 2003-06-09
Points: 0

from http://java.sun.com/developer/technicalArticles/WebServices/become/
Become.com's innovative shopping search engine has done what many in the search engine community thought impossible: The company has successfully created a Java technology web crawler that may be the most sophisticated, massively scaled Java technology application in existence, obtaining information on over 3 billion web pages and writing well over 8 terabytes of data (and growing) on 30 fully distributed servers in seven days.