Robots Retrieving the entire World Wide Web

Assuming a robot can find all documents on the Web, the speed of the network may prevent the robot from doing this in a reasonable amount of time:

• Several robots (Lycos and Alta Vista for instance) have already found that the Web contains over 20 million documents, for a total of over 100 gigabytes of textual information.
  
  
• Network speed varies a lot on Internet. At 3 kbyte/sec loading 100 gig would take over 1 year. (This is the best possible result with an 28.8 modem.) At 1.5 mbits/sec, a full T1 line, loading 100 gig would take 8 days.
  
  
• At 1 document per second loading the entire Web would take 8 months. To achieve this rate at least 10 documents must be loaded in parallel. To fully load a T1 line at least 250 documents must be loaded in parallel.

There is no standard way to know when a document becomes obsolete, other than to try to download it again and check whether it still exists. A conditional-get can be used to avoid the actual download if the document is not modified. Even the largest search tools like Lycos and Alta Vista need a few weeks between visits to the same documents.

There is no standard way to know when and where new documents are created, other than to try to download the whole Web over and over and look for new links. This means that the robots of major search tools take a long time to discover new documents.