robots.txt
standard used to advise web crawlers and scrapers not to index a web page or site
Press Enter · cited answer in seconds
0 sources
robots.txt
Summary
robots.txt is a computer network protocol[1]. robots.txt ranks in the top 5% of computer_network_protocol entities by monthly Wikipedia readership (966 views/month).[2]
Key Facts
- robots.txt's image is recorded as Robots txt.svg[3].
- robots.txt's instance of is recorded as computer network protocol[4].
- robots.txt's instance of is recorded as file format[5].
- robots.txt's instance of is recorded as technical standard[6].
- robots.txt's Commons category is recorded as Robots.txt[7].
- robots.txt's Freebase ID is recorded as /m/0psn8[8].
- robots.txt's official website is recorded as https://www.robotstxt.org/[9].
- robots.txt's described at URL is recorded as https://tools.ietf.org/html/draft-giudici-web-robots-cntrl-00[10].
- robots.txt's described at URL is recorded as https://tools.ietf.org/html/draft-koster-rep-04[11].
- robots.txt's described at URL is recorded as https://wiki.archiveteam.org/index.php/Robots.txt[12].
- robots.txt's described by source is recorded as RFC 9309: Robots Exclusion Protocol[13].
- robots.txt's Stack Exchange tag is recorded as https://stackoverflow.com/tags/robots.txt[14].
- robots.txt's Stack Exchange tag is recorded as https://webmasters.stackexchange.com/tags/robots.txt[15].
- robots.txt's File Format Wiki page ID is recorded as Robots.txt[16].
- robots.txt's File Format Wiki page ID is recorded as Robots_Exclusion_Standard[17].
- robots.txt's Quora topic ID is recorded as Robots-Exclusion-Protocol[18].
- robots.txt's Quora topic ID is recorded as robots-txt[19].
- robots.txt's Zhihu topic ID is recorded as 19600309[20].
- robots.txt's Microsoft Academic ID is recorded as 163978704[21].
- robots.txt's Mozilla Developer Network article is recorded as Glossary/Robots.txt[22].
- robots.txt's FOLDOC ID is recorded as robots.txt[23].
- robots.txt's ComputerLanguage.com definition is recorded as robots.txt[24].
Why It Matters
robots.txt ranks in the top 5% of computer_network_protocol entities by monthly Wikipedia readership (966 views/month).[2] robots.txt has Wikipedia articles in 22 language editions, a strong signal of global cultural recognition.[25] robots.txt is known by 10 alternative names across languages and contexts.[26]