What is robots.txt for AI Agents?
Also called Robots Exclusion Protocol, AI crawler controls, robots.txt.
robots.txt is a text file at a website's root that tells automated clients which paths they may request, standardized as the Robots Exclusion Protocol. Applied to AI, site owners add rules naming the crawlers operated by AI companies in order to allow or refuse them. Compliance is voluntary, so the file expresses a request rather than an enforced control.
The file lists client names followed by disallow and allow rules for path prefixes, and a well behaved crawler fetches it before requesting anything else. AI companies publish the names their crawlers send, and many separate them by purpose, using distinct names for bulk collection and for fetches made in response to one user's specific request. That separation lets a site refuse one while permitting the other.
Nothing enforces the file. A client that ignores it faces no technical obstacle, so operators who want enforcement rely on authentication, rate limits, or edge blocking that rejects requests by network signature instead. The file also governs only whether content may be fetched. It says nothing binding about what may be done with that content afterwards, which is the question most publishers actually care about.
Whether an agent acting for a single person is a crawler at all is genuinely unsettled. Fetching one page because a user asked for it resembles that person opening a browser more than it resembles bulk collection, and vendors and publishers draw the line differently. Standards work is underway to express content usage preferences more precisely, and practice is still changing.
For site owners, a workable posture is to state rules explicitly rather than rely on defaults, to remember that blocking every automated client also removes a site from AI generated answers, and to pair the file with real enforcement where it matters. For agent builders, honoring the file, identifying yourself honestly, and limiting request rates is both the norm and the way to avoid being blocked outright.
Key points
- Root level file listing which paths automated clients may request.
- Standardized as the Robots Exclusion Protocol, with voluntary compliance.
- AI vendors publish crawler names, often split by purpose.
- Controls fetching only, not what happens to retrieved content.
- Whether a user triggered agent counts as a crawler is unsettled.
In practice
A publisher wants search visibility but no bulk collection for model training. Their robots.txt allows general search crawlers everywhere, disallows the named training crawlers from the article directory, and leaves the marketing pages open to everything. Because the file is only a request, they also add rate limiting at their edge, which is what actually stops a client that chooses to ignore the rules.