robots.txt
standard used to advise web crawlers and scrapers not to index a web page or site
Facts
3 properties- Instance of
- computer network protocol, file format, technical standard[1][2]
- Official website
- www.robotstxt.org[3]
- Described by source
- RFC 9309: Robots Exclusion Protocol[4]
🏷️ Also known as
en Robots Exclusion Protocol
en robots.txt protocol
en robots.txt
ar معيار استبعاد الروبوتات
da Robot Exclusion Standard
de robots.txt
de Robots Exclusion Standard
de Robots Exclusion Protocol
es Estandar de exclusion de robots
es Estándar de exclusión de robots
fa فایل robots.txt
fi Robotin rajausstandardi
fr protocole d'exclusion des robots
fr robots.txt
he robots.txt
he פרוטוקול אי הכללת רובוטים
it protocollo di esclusione robot
ja robots.txt
ko 로봇 배제 표준
nl robots.txt
🔖 Identifiers · 11
- ComputerLanguage.com definition
- robots.txt
- Described at URL
- tools.ietf.org/…/draft-giudici-web-robots-cntrl-00, tools.ietf.org/…/draft-koster-rep-04, wiki.archiveteam.org
- FOLDOC ID
- robots.txt
- File Format Wiki page ID
- Robots.txt, Robots_Exclusion_Standard
- Freebase ID
- /m/0psn8
- GitLab.com topic ID
- robots.txt
- Microsoft Academic ID (discontinued)
- 163978704
- Mozilla Developer Network article
- Glossary/Robots.txt
- Quora topic ID
- Robots-Exclusion-Protocol, robots-txt
- Stack Exchange tag
- stackoverflow.com, webmasters.stackexchange.com
- Zhihu topic ID
- 19600309
🔗 Connections
Instance of 3
← FactGrid item ID 1
Press Enter · cited answer in seconds