= web crawler

olemus
netirobot, mis sirvib süstemaatiliselt WWW veebilehti,
enamasti nende sisu automaatseks indekseerimiseks,
arvestades ka veebiserveri juhiseid failis robots.txt

Wiktionary:
(Internet) a software program that gathers specific information in an automated and orderly way from the Internet

ülevaateid
https://en.wikipedia.org/wiki/Web_crawler

https://www.woorank.com/en/blog/how-a-crawler-works-back-to-the-basics

https://nlp.stanford.edu/IR-book/pdf/20crawl.pdf

https://iopscience.iop.org/article/10.1088/1742-6596/1449/1/012036/pdf

https://ijarcce.com/wp-content/uploads/2016/02/IJARCCE-52.pdf