MOMspider
is developed and maintained by
Roy T. Fielding
from the University of California at Irvine. It is a tool for
Maintaining Distributed Hypertext Infostructures.
- MOMspider uses an instruction file which lists
the documents from where to start, and the user who should be notified
about the findings..
The start-document can be a user's hotlist.
By running MOMspider from cron at night, all users' hotlists can be
verified, and users notified of changes and deletions.
- MOMspider will traverse remote servers slowly, so as to not upset
Webmasters. (These speed limits do not apply when searching through the
local server.)
- MOMspider checks the /robots.txt file on every host,
to avoid retrieving documents the Webmasters don't want to give to robots.
