olemus
otsingusse võtu keeld failis robots.txt
= exclusion-from-search directive in the robot.txt file

ülevaateid
https://technicalseo.com/crawl-indexation/directives/robots-txt/

http://www.studyzone4u.com/post-details/what-is-robotstxt-file

https://en.wikipedia.org/wiki/Robots_exclusion_standard