olemus
protokoll otsirobotitele teatamiseks, milliseid
veebisaidi osi soovib omanik otsingust välja jätta,
kasutades teatamiseks faili robots.txt;
faktiline standard aastast 1994
= a protocol for telling search robots which parts of the website the owner wants to exclude from the search, using the robots.txt file for this purpose

ülevaateid
https://en.wikipedia.org/wiki/Robots_exclusion_standard

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt