Robots.txt Generator and Tester
Start from a preset such as Allow everything or Block AI training crawlers, add the paths each crawler should avoid and your sitemap, then upload the file to your site's root as /robots.txt. The tester shows whether Googlebot, Bingbot or any bot may fetch a URL, and which rule decides. Blocking crawling does not stop indexing.
Updated
How to use the Robots.txt Generator and Tester
- 1Start from a preset, such as Allow everything or Block AI training crawlers.
- 2Add the paths each crawler should stay out of, and your sitemap address.
- 3Copy or download the file and upload it to the root of your site as /robots.txt.
- 4Switch to Test URLs, choose a crawler and enter addresses to see whether each is allowed.
How it works
A robots.txt file is a set of groups. Each group names one or more crawlers in User-agent lines and lists Allow and Disallow rules for them. A crawler follows only the groups that name it, merged together, and falls back to the User-agent: * group only when no group names it — so a Googlebot group replaces the * rules for Googlebot rather than adding to them.
When several rules match a URL, the most specific — the rule with the longest path — wins, and when an Allow and a Disallow are equally long, Allow wins. An asterisk matches any run of characters and a dollar sign marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF but not /report.pdf?download=1. The tester applies exactly these rules, from RFC 9309, which Google and Bing follow.
robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it; to keep a page out of search, let it be crawled and give it a noindex robots meta tag. And it is a request, not a lock: reputable crawlers honour it, but it does not stop anyone determined to fetch your pages.
The AI crawler preset adds a group for the tokens AI companies publish for their training and answer-engine crawlers. Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: blocking them opts content out of AI training without affecting Google Search or Apple's search features.
Tips and common mistakes
- Test your important pages with the URL tester before uploading the file.
- Do not block the CSS and JavaScript files your pages need; search engines use them to render the page.
- Add your sitemap address to the file so crawlers can find it.
- To keep a page out of search results, use a noindex tag and let it be crawled; blocking it in robots.txt stops crawlers seeing the noindex.