Skip to content
Utilboxes

Robots.txt Generator and Tester

Build a robots.txt file from simple settings, or paste an existing one and test exactly which URLs each crawler may fetch — and which rule decides it.

Runs entirely in your browser

How to use the Robots.txt Generator and Tester

  1. 1Start from a preset, such as Allow everything or Block AI training crawlers.
  2. 2Add the paths each crawler should stay out of, and your sitemap address.
  3. 3Copy or download the file and upload it to the root of your site as /robots.txt.
  4. 4Switch to Test URLs, choose a crawler and enter addresses to see whether each is allowed.

How it works

A robots.txt file is a set of groups. Each group names one or more crawlers in User-agent lines and lists Allow and Disallow rules for them. A crawler follows only the groups that name it, merged together, and falls back to the User-agent: * group only when no group names it — so a Googlebot group replaces the * rules for Googlebot rather than adding to them.

When several rules match a URL, the most specific — the rule with the longest path — wins, and when an Allow and a Disallow are equally long, Allow wins. An asterisk matches any run of characters and a dollar sign marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF but not /report.pdf?download=1. The tester applies exactly these rules, from RFC 9309, which Google and Bing follow.

robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it; to keep a page out of search, let it be crawled and give it a noindex robots meta tag. And it is a request, not a lock: reputable crawlers honour it, but it does not stop anyone determined to fetch your pages.

The AI crawler preset adds a group for the tokens AI companies publish for their training and answer-engine crawlers. Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: blocking them opts content out of AI training without affecting Google Search or Apple's search features.

Frequently asked questions

Where do I put robots.txt?
At the root of each host, so it is served from https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.
Does Disallow remove a page from Google?
No. It stops Googlebot fetching the page, but the address can still be indexed from links. Put a noindex meta tag on a crawlable page to keep it out of results.
Will blocking AI crawlers stop my content being used by AI?
It stops crawlers that respect robots.txt from collecting new content. It does not remove anything already collected, and crawlers that ignore the file are not affected.
Why is Crawl-delay flagged?
Google ignores Crawl-delay and adjusts its crawl rate automatically based on how your server responds. Bing and Yandex do honour it.

Related tools

Open Graph PreviewPreview how a link looks when shared, and check or write its Open Graph tags.Developer ToolsSSL Certificate CheckerSee when a domain's SSL certificate expires and whether its chain is trusted.Developer ToolsFavicon GeneratorMake favicon.ico, Apple touch and Android icons from an image or letter.Image ToolsURL EncoderPercent-encode a whole URL or a single query parameter.Developer Tools