How to write a robots.txt file — with examples

A robots.txt file is a plain text file at /robots.txt that tells crawlers which paths they may fetch, using User-agent, Disallow and Allow lines. Copy one of the examples below, or create a robots.txt file with the generator and test your URLs against it. A single wrong line can stop search engines crawling an entire site, so check it before uploading.

robots.txt examples to copy

Allow every crawler to fetch everything, and list the sitemap:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Keep every crawler out of one folder:

User-agent: *
Disallow: /admin/

Block one crawler completely and leave the rest alone:

User-agent: BadBot
Disallow: /

User-agent: *
Disallow:

Ask AI training crawlers to stay out while search engines carry on as normal:

User-agent: GPTBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Block the whole site (for a staging copy only — never on a live site):

User-agent: *
Disallow: /

Replace example.com with your own domain. Several User-agent lines in a row share the rules beneath them.

Where it goes

At the root of each host: https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain, such as shop.example.com, needs its own. The file name must be in lower case.

How the file is structured

The file is a set of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for paths. A simple file that lets everyone in, keeps them out of an admin area and lists the sitemap:

• User-agent: *
• Disallow: /admin/
• Sitemap: https://example.com/sitemap.xml

The asterisk group applies to every crawler not named elsewhere. An empty Disallow: line allows everything.

How rules are matched

• A crawler follows only the groups that name it. If there is a group for Googlebot, Googlebot ignores the asterisk group entirely — it does not add the two together.
• The most specific rule wins — the one with the longest matching path. If an Allow and a Disallow are equally long, Allow wins.
• Paths are case-sensitive. Disallow: /Private does not block /private.
• Wildcards: * matches any run of characters and $ marks the end of the address. Disallow: /*.pdf$ blocks addresses ending in .pdf but not /report.pdf?download=1.

These are the rules of the robots exclusion standard, RFC 9309, which Google and Bing follow.

Build and test a file

1. Open the Robots.txt Generator and Tester and start from a preset.
2. Add the paths each crawler should stay out of, and your sitemap address.
3. Switch to Test URLs, choose a crawler and enter some of your important addresses. The tester shows whether each is allowed and which rule decided it.
4. Copy or download the file and upload it to the root of your site.

Blocking AI training crawlers

Several AI companies publish the names their crawlers use, such as GPTBot and Google-Extended, and say they respect robots.txt. A group naming them with Disallow: / asks them not to collect your content. Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: blocking them opts content out of AI training without affecting Google Search or Apple's search features. Blocking does not remove anything already collected, and crawlers that ignore the file are not affected. The generator's AI crawler preset adds a group for the published names.

robots.txt does not remove pages from search

This is the most common misunderstanding. Disallow stops a crawler fetching a page, but the address can still appear in search results if other sites link to it — just without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag. Blocking it in robots.txt would stop search engines seeing the noindex.

Nor is robots.txt security. It is a public request that reputable crawlers honour; anyone can read it, and it can point curious people straight at the paths you listed. Protect private areas with logins.

Mistakes that hide a site

• Leaving Disallow: / from a staging site on the live site — it blocks everything.
• Blocking CSS and JavaScript files. Search engines need them to render the page.
• Relying on Crawl-delay for Google. Google ignores it and sets its own pace; Bing and Yandex do honour it.

Related

Once crawlers can reach your pages, the rest is about what they find there: structured data from the Schema Markup Generator and sharing tags checked with the Open Graph Preview.