Below is a MOMspider
instruction file:
# Instructions file for MOMspider # The file with hosts to avoid is here: AvoidFile /home/debra/.momspider-avoid # The file with sites, written by momspider SitesFile /home/debra/.momspider-sites # How many days between checks of a site's /robots.txt file? SitesCheck 1 # Who should Webmasters send a report to when something goes wrong ? ReplyTo debra@win.tue.nl # MaxDepth is to prevent infinite holes MaxDepth 10 <Tree Name WWW TopURL http://wwwis.win.tue.nl/~debra/ IndexURL http://pcpaul.win.tue.nl/win-index.html IndexFile /usr/local/etc/httpd/htdocs/win-index.html IndexTitle MOMspider Index for homepage on www.win EmailAddress debra@win.tue.nl EmailBroken EmailRedirected EmailChanged 7 EmailExpired 7 Exclude http://www.win.tue.nl:8080/ >