Crawl downloading is the backup download strategy. Unlike the the Nici-Quick download strategy, Crawl downloading systematically downloads and searches through all links. Before starting a Crawl download, a number of parameters need to be chosen. More
The activity of using software to recursively download web documents by following links. There are a variety of crawl methods, including: focused crawl, smart crawl, incremental crawl, targeted crawl, and customized crawl. See also crawler.