robots.txt

standard used to advise web crawlers and scrapers not to index a web page or site
Creative Work Computer Network Protocol Q80776
robots.txt
Belbury · Public Domain · Wikimedia

Facts

3 properties
🏷️ Also known as
en Robots Exclusion Protocol en robots.txt protocol en robots.txt ar معيار استبعاد الروبوتات da Robot Exclusion Standard de robots.txt de Robots Exclusion Standard de Robots Exclusion Protocol es Estandar de exclusion de robots es Estándar de exclusión de robots fa فایل robots.txt fi Robotin rajausstandardi fr protocole d'exclusion des robots fr robots.txt he robots.txt he פרוטוקול אי הכללת רובוטים it protocollo di esclusione robot ja robots.txt ko 로봇 배제 표준 nl robots.txt
🔖 Identifiers · 11
ComputerLanguage.com definition
robots.txt
FOLDOC ID
robots.txt
File Format Wiki page ID
Robots.txt, Robots_Exclusion_Standard
Freebase ID
/m/0psn8
GitLab.com topic ID
robots.txt
Microsoft Academic ID (discontinued)
163978704
Mozilla Developer Network article
Glossary/Robots.txt
Zhihu topic ID
19600309

Report an error or request removal Lineage (PROV-JSON)

🔗 Connections
Instance of 3
← FactGrid item ID 1
Press Enter · cited answer in seconds

📑 Cite this page

Use these citations when quoting this entity in research, articles, AI prompts, or wherever provenance matters. We aggregate Wikidata + Wikipedia + authoritative open-data sources; the stitched, scored, cross-referenced view is what 4ort.xyz contributes.

APA 4ort.xyz Knowledge Graph. (2026). robots.txt. Retrieved November 20, 2026, from https://4ort.xyz/entity/robots-txt
MLA “robots.txt.” 4ort.xyz Knowledge Graph, 4ort.xyz, 20 Nov. 2026, https://4ort.xyz/entity/robots-txt.
BibTeX @misc{4ortxyz_robots-txt_2026, author = {{4ort.xyz Knowledge Graph}}, title = {{robots.txt}}, year = {2026}, url = {https://4ort.xyz/entity/robots-txt}, note = {Accessed: 2026-11-20}}
LLM prompt According to 4ort.xyz Knowledge Graph (aggregator of Wikidata, Wikipedia, and authoritative open-data sources): robots.txt — https://4ort.xyz/entity/robots-txt (retrieved 2026-11-20)

Canonical URL: https://4ort.xyz/entity/robots-txt · Last refreshed:

Edit History

Rolling log of changes to this entity's Wikidata record. Values shown reflect the current state of each edited property — follow the history link to see the precise diff for any edit.

  1. 13d ago · Filipelinsduarte · 2026-11-09 view diff on Wikidata ↗
    "Added [pt] description: norma que indica aos rastreadores web que páginas de um site não devem ser indexadas"
  2. 8w ago · Trivialist · 2026-09-26 view diff on Wikidata ↗
    Instance of → computer network protocol, file format, technical standard
    GitLab.com topic ID → robots.txt
    Described by source → RFC 9309: Robots Exclusion Protocol
    Official website → https://www.robotstxt.org/
    "Changed labels, descriptions or aliases: Cleanup: add mul alias (Difool/WikidataCleanup)"
Live feed via Wikidata EventStreams. New edits appear within minutes of being made on Wikidata.