= web crawler

olemus
netirobot:
sirvib süstemaatiliselt WWW veebilehti,
enamasti nende sisu automaatseks indekseerimiseks,
arvestades ka veebiserveri juhiseid failis robots.txt

Wiktionary:
(Internet) a software program that gathers specific information in an automated and orderly way from the Internet

ülevaateid
https://en.wikipedia.org/wiki/Web_crawler

https://www.woorank.com/en/blog/how-a-crawler-works-back-to-the-basics

http://ssrg.site.uottawa.ca/docs/CASCON2013.pdf

https://nlp.stanford.edu/IR-book/pdf/20crawl.pdf

https://iopscience.iop.org/article/10.1088/1742-6596/1449/1/012036/pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.4776&rep=rep1&type=pdf

https://ijarcce.com/wp-content/uploads/2016/02/IJARCCE-52.pdf