The best advice for future robot writers is: don't !
Look for existing search engines and robots.
Chances are there will be a robot or engine that already does what you want.
Below is some advice if you do write your own robot:
- Identify your robot
- Provide the HTTP User-agent field to identify the
robot.
- Be traceable
- Make sure your system can be found through the IP protocol.
Do not run through a SLIP or PPP connection with dynamic IP numbers.
Provide a valid HTTP Referer field.
- Identify yourself
- Provide your email address through the HTTP From field.
- Announce your robot
- Robots are announced through Usenet Netnews, in
comp.infosystems.www.providers, and through the
robot mailing list.
Also send email to Webmasters of sites you expect to be affected,
and to your own system administrators.
- Be there
- Do not test a new robot during your absence, at night or during the
weekend.
Be present to respond to email from people who experience problems
caused by your robot.
-
