Robots.txt

Generate robots.txt with presets (including blocking AI crawlers), validate an existing file, and test which URLs each crawler can access using Google's RFC 9309 matching rules (longest match wins, * and $ wildcards).

Presets

Group 1
User-agents
*
Rules
Sitemaps
robots.txt 
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/

Sitemap: https://example.com/sitemap.xml

robots.txt controls crawling, not indexing. Blocked URLs can still appear in search results if linked elsewhere — use noindex for that. It is also not access control: well-behaved bots honor it, others ignore it.