Robots.txt Generator

Build and download your robots.txt file for free online

Quick Presets
Bot Rules
Default Policy
Sitemap URL
Crawl Delay
0 (none)
Options
Allow image crawlers
Googlebot-Image, Bingbot-Image
Block AI training bots
GPTBot, Claude-Web, CCBot, PerplexityBot
Block social media crawlers
Twitterbot, facebookexternalhit, LinkedInBot