Robots.txt Generator — Free Online
Generate a robots.txt file for your website. Select crawlers, set allow/disallow rules, add sitemap URL, and download the file. Free, no signup required.
Select Crawlers
Rules for *
User-agent: * Allow: /
How it works
Enter your data
Fill in the form fields or paste your content to get started.
Processed in your browser
Everything runs client-side — your data never leaves your machine.
Copy or download results
Get instant results you can copy to clipboard or download.
Features
Crawler Presets
Quick toggles for Googlebot, Bingbot, GPTBot, CCBot, and more.
Allow/Disallow Rules
Add custom rules per crawler with common path suggestions.
Sitemap Support
Add your sitemap URL to help crawlers discover all your pages.
Live Preview
See the generated robots.txt update in real-time.
Download File
Download the generated robots.txt file ready to upload.
AI Crawler Blocking
Easy toggles to block AI training crawlers.
Frequently asked questions
What is robots.txt?
robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages or sections they should or shouldn't access. It's the first file crawlers check when visiting your site.
Does robots.txt block pages from appearing in Google?
Not exactly. Robots.txt prevents crawling, but pages can still appear in search results if other sites link to them. To prevent indexing, use the 'noindex' meta tag instead.
What is the Crawl-delay directive?
Crawl-delay tells crawlers to wait a specified number of seconds between requests. Google ignores this directive (use Google Search Console instead), but Bing and others respect it.
Should I block AI crawlers?
You can block AI crawlers like GPTBot (OpenAI), CCBot (Common Crawl), and Google-Extended if you don't want your content used for AI training. This is increasingly common for content publishers.
Where do I put the robots.txt file?
Place robots.txt in the root directory of your website (e.g., https://example.com/robots.txt). It must be accessible at this exact URL for crawlers to find it.
Related tools
Meta Tag Generator — Free Online SEO Tool
Generate HTML meta tags for SEO including title, description, keywords, robots, canonical URL, viewport, and Open Graph tags. Free online, no signup required.
Open Graph Meta Tag Generator — Free Online
Generate Open Graph and Twitter Card meta tags for your website. Preview how your page will look when shared on social media. Free, no signup required.
Google SERP Preview — Free Online Simulator
Preview exactly how your page will appear in Google search results. Check title and description length, pixel width, and optimize for maximum click-through rate. Free, no signup.
Use cases
SEO Management
Control how search engines crawl your site to optimize crawl budget and prevent indexing of duplicate content.
AI Crawler Blocking
Block AI training crawlers like GPTBot and CCBot from scraping your content.
Development Sites
Prevent search engines from indexing staging and development environments.