How matching works
A crawler uses the group whose User-agent matches it most specifically, falling back to *. Within the group, the rule with the longest matching path wins; on a tie, Allow beats Disallow. * matches any characters and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it; use a noindex meta tag on a crawlable page to keep it out. Never block CSS or JavaScript that pages need to render.
AI crawlers
The “Block AI crawlers” preset adds a group for the user agents that AI companies document as honouring robots.txt, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot. Blocking Google-Extended does not affect Google Search.
Questions
Where does robots.txt go?
At the root of each host: https://example.com/robots.txt. Subdomains need their own file.
Should robots.txt list my sitemap?
Yes. A Sitemap: line with the absolute URL lets every search engine discover it, not only those where you submitted it.
Does Google respect Crawl-delay?
No. Bing and Yandex do. Google adjusts its crawl rate automatically.
Will Disallow remove a page from Google?
Not reliably. Allow crawling and add noindex, or remove the page and return 404/410.