Robots Finding the entire World Wide Web

Assuming network bandwidth is not a problem, robots still have problems finding all documents on the Web:

• The RBSE spider project has found that to 59% of all documents found there is only one link (and to 96% there are less than 5 links). It is unknown to how many documents there are no links.
  
  
• It is unknown which, or how many, starting points are needed to be able to reach the entire Web from them.
  
  
• The Web may contain documents that can only be reached by clicking in an image. There are too many coordinates for a robot to try all of them.
  
  
• The Web may contain documents that can only be reached by filling out a form.

Documents that can only be reached by clicking in an image should soon disappear. More and more authors follow the guideline that says that in addition to a clickable image there should also be a text-based menu leading to the same documents.

Documents that can only be reached by filling out a form may be considered part of the Web, but in many cases they are not really necessary or even useful for robots. When they offer "real" documents these should also be made available without using the form. Other "documents" resulting from filling out a form may be a confirmation of an order, ranging from a pizza to a nuclear missile, may send someone a (useless) e-mail message, or may register the robot for a conference or some other event.