robots.txt
standard used to advise web crawlers and scrapers not to index a web page or site
Press Enter · cited answer in seconds
0 sources
robots.txt
Summary
robots.txt is a computer network protocol[1]. robots.txt ranks in the top 5% of computer_network_protocol entities by monthly Wikipedia readership (11,561 views/month).[2]
Key Facts
- robots.txt's instance of is recorded as computer network protocol[3].
- robots.txt's instance of is recorded as file format[4].
- robots.txt's instance of is recorded as technical standard[5].
- robots.txt's Commons category is recorded as Robots.txt[6].
- robots.txt's official website is recorded as https://www.robotstxt.org/[7].
- robots.txt's described at URL is recorded as https://tools.ietf.org/html/draft-giudici-web-robots-cntrl-00[8].
- robots.txt's described at URL is recorded as https://tools.ietf.org/html/draft-koster-rep-04[9].
- robots.txt's described at URL is recorded as https://wiki.archiveteam.org/index.php/Robots.txt[10].
- robots.txt's described by source is recorded as RFC 9309: Robots Exclusion Protocol[11].
- robots.txt's Stack Exchange tag is recorded as https://stackoverflow.com/tags/robots.txt[12].
- robots.txt's Stack Exchange tag is recorded as https://webmasters.stackexchange.com/tags/robots.txt[13].
Why It Matters
robots.txt ranks in the top 5% of computer_network_protocol entities by monthly Wikipedia readership (11,561 views/month).[2] robots.txt has Wikipedia articles in 22 language editions, a strong signal of global cultural recognition.[14] robots.txt is known by 24 alternative names across languages and contexts.[15]