MOMspider
can be configured by parameters, either on the command line or in the
instruction file. (We present the instruction file only.)
- AvoidFile pathname
- Name for the user's file with sites and documents to avoid.
- SitesFile pathname
- Name for the user's file with sites and dates for revisiting them.
- SitesCheck N
- The number of days between checks of a site's /robots.txt
file.
- ReplyTo email address
- Must contain the email address of the person running MOMspider.
This address is given to all visited servers as the person to contact
when something goes wrong.
- MaxDepth N
- Maximum depth of any traversal. This prevents MOMspider from getting
stuck in infinite recursive loops.
