Definitions for "Web crawler"
A bot that searches the world wide web for new and updated web pages. Found pages are categorized by subject and placed in a database. Typically, an associated search engine will access that database.
Service that scans web documents and adds them to a database. After having indexed one page it follows all links and indexes them as well. See also search engine.
a computer program that operates across the Internet to obtain information from the Web sites of others