SiteMapperPro

Robots.txt generator

Build a robots.txt that blocks only what it should, points every crawler at your sitemap, and optionally opts out of AI training crawlers.

Allow wins over Disallow when it is the longer (more specific) match.

Block these AI crawlers entirely

Extra groups for specific crawlers

Your robots.txt

# robots.txt generated with sitemapperpro.io

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Test specific URLs against it in the robots.txt tester.

How crawlers read the file

Under RFC 9309, a crawler picks the one group whose User-agent best matches its name, falling back to *, and ignores every other group. That is why the AI-crawler groups above stand alone: Googlebot never reads them, and GPTBot never reads your * rules. Inside the chosen group, the most specific (longest) matching path wins, and if an Allow and a Disallow tie, the Allow wins.

The Sitemap: line sits outside all groups and applies to everyone. It must be an absolute URL, and it may point to a sitemap index. The sitemaps.org protocol also lets a robots.txt reference a sitemap on another host, which is how sites with a CDN-hosted sitemap prove ownership.

Wildcards

  • * matches any sequence of characters: Disallow: /*?sort= blocks any URL with a sort parameter.
  • $ anchors the end: Disallow: /*.pdf$ blocks PDFs but not /pdf-guide.
  • Paths are case-sensitive: /Admin/ does not block /admin/.

Questions

What should a basic robots.txt contain?

For most sites: a User-agent: * group that disallows only genuinely private or useless areas (cart, internal search results, admin), and a Sitemap: line with the full URL of your sitemap or sitemap index. An empty Disallow: means everything is allowed.

Does Disallow remove a page from Google?

No. robots.txt controls crawling, not indexing. A disallowed URL can still be indexed without its content if other pages link to it. To keep a page out of search results, allow crawling and use a noindex meta tag or header, or password-protect it.

Should I block AI crawlers?

That is a business decision. Several AI companies publish user-agent tokens (for example GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) and say they honour robots.txt. The generator adds a group for each one you tick.

Where do I put the file?

At the root of each host, e.g. https://www.example.com/robots.txt. It applies only to that protocol, host and port, so subdomains need their own file.

Sources