# robots.txt

> standard used to advise web crawlers and scrapers not to index a web page or site

**Wikidata**: [Q80776](https://www.wikidata.org/wiki/Q80776)  
**Wikipedia**: [English](https://en.wikipedia.org/wiki/Robots.txt)  
**Source**: https://4ort.xyz/entity/robots-txt


## References

1. Freebase Data Dumps. 2013
2. Quora