robots.txt Generator
Create a correct robots.txt file: allow or block crawlers, disallow private paths, add your sitemap and block AI crawlers such as GPTBot. Then test any URL against the rules and see which line decides it.
How to Use
- Pick a template: Allow all, Block all or Custom rules, or press a preset.
- For Custom, list the paths to disallow (one per line), any exceptions to allow, and an optional crawl delay.
- Add your sitemap URL and tick any AI or scraper bots you want to keep out.
- Under Test a URL, type a crawler name and a path or full URL to see whether it is allowed and which line decides it.
- Read Show Work for why each line is there and how the test was matched, then Copy or Download the file and put it at the root of your site.
Worked Example
A WordPress site uses this file:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
Googlebot asks for /wp-admin/admin-ajax.php. There is no Googlebot group, so the * group applies. Both rules match: Disallow: /wp-admin/ is 10 characters long and Allow: /wp-admin/admin-ajax.php is 24. The longest match wins, so the URL is allowed.
Googlebot asks for /wp-admin/options.php. Only Disallow: /wp-admin/ matches, so it is blocked. /blog/ matches neither rule, and anything no rule matches is allowed.
A tie. With Disallow: /page and Allow: /page, the URL /page matches both at 5 characters each. Google and RFC 9309 take the less restrictive rule, so it is allowed. Which line comes first does not matter.
The common mistake: Disallow: / where Disallow: was meant. The empty value matches nothing and allows the whole site; the single slash matches every path and blocks the whole site. It is one character, and it is how staging files copied to a live server take sites out of search. A close second is Disallow: /admin with no trailing slash: rules are prefix matches, so it also blocks /administrator-guide and /admin-tips. Use /admin/.
Show Work
Syntax Rules
Where robots.txt Came From
Martijn Koster proposed the Robots Exclusion Protocol in February 1994 on the www-talk mailing list, after early web crawlers had hammered servers with requests. It had no standards body behind it, yet within months every serious crawler checked /robots.txt first, and it stayed an informal agreement for 25 years. Each search engine added its own extensions — Allow, the * and $ wildcards, Crawl-delay, Sitemap — and they did not always agree on the details.
In July 2019 Google proposed making it a formal standard and open-sourced the parser Googlebot uses. The IETF published it as RFC 9309 in September 2022. It standardises User-agent, Allow and Disallow, the wildcards, the longest-match rule, and a minimum of 500 KiB that crawlers must read; Crawl-delay and Sitemap remain extensions. Since 2023 the file has also become the main way to opt out of AI training crawlers such as GPTBot and Google-Extended.
About This Tool
This tool writes a robots.txt from a template, your own Allow and Disallow paths, a crawl delay, a sitemap URL and a list of AI crawlers to block. Paths without a leading slash get one, so a typo cannot produce a rule that matches nothing.
It also tests the file it built. Type a crawler name and a path or full URL, and it picks the group that crawler would use, checks every rule with the * and $ wildcards, and applies the longest-match and Allow-wins-a-tie rules from RFC 9309, highlighting the line that decided. Everything runs in your browser; nothing you type is uploaded.
It is for site owners launching or moving a site, developers checking a rule before it goes live, and anyone who wants to see why a page is or is not being crawled.
Related tools: Meta Tag & Open Graph Generator, .htaccess Generator, and Schema / JSON-LD Generator.
Frequently Asked Questions
Where does the robots.txt file go?
At the root of the host, so it loads at https://example.com/robots.txt. Crawlers look there and nowhere else: a file at /blog/robots.txt is ignored, and each subdomain (shop.example.com, www.example.com) needs its own. Google reads at most the first 500 KiB of the file.
Does Disallow hide a page from search results?
No. Disallow only stops a well-behaved crawler fetching the page. If other sites link to it, Google can still list the bare URL with no description. To keep a page out of results, let it be crawled and add <meta name="robots" content="noindex"> — a blocked page’s noindex is never seen. And robots.txt is public, so never list secret paths in it.
When Allow and Disallow both match, which one wins?
The rule with the longest path wins, counted in characters. For /wp-admin/admin-ajax.php, Allow: /wp-admin/admin-ajax.php (24 characters) beats Disallow: /wp-admin/ (10), so it is allowed. If two matching rules are the same length, Google and RFC 9309 pick the less restrictive one, so Allow wins the tie. The order of the lines makes no difference.
Should I block AI crawlers?
That is a choice about your content. GPTBot (OpenAI), CCBot (Common Crawl), ClaudeBot (Anthropic) and Google-Extended (Gemini training) collect text for AI models; blocking them asks them not to. Google-Extended does not affect Google Search ranking. Reputable crawlers honour robots.txt, but it is a request, not an enforced rule.
Does Googlebot obey Crawl-delay?
No. Google ignores Crawl-delay and sets its own rate from how fast your server responds. Bing and Yandex do honour it: Crawl-delay: 10 asks them to wait 10 seconds between requests, which is at most 8,640 pages a day — too slow for a large site.
How do I use the robots.txt Generator?
Simply pick your options and read the result, which refreshes the instant you change something. There is nothing to submit and nothing to wait for.
Does it cost anything or need an account?
No. The tool is completely free, there is no account to create, and it keeps working offline after the page first loads.
Is anything I type uploaded?
No. The tool works entirely on your device, so the values you enter never leave your browser.
Common Use Cases
Launch a new site
Two lines — User-agent: * and an empty Disallow: — plus a Sitemap line let every crawler in and point them at all your pages from day one.
WordPress admin
Disallow /wp-admin/ but Allow /wp-admin/admin-ajax.php, which themes call from public pages. The tester shows the 24-character Allow beating the 10-character Disallow.
Shop filters and search
Stop crawlers wasting time on endless /search and ?sort= variations: Disallow: /*?sort= blocks /shoes?sort=price but not /shoes.
Opt out of AI training
Give GPTBot, CCBot, ClaudeBot and Google-Extended their own group with Disallow: /, while search engines keep the * rules.
Staging servers
Disallow: / keeps crawlers off a test copy of a site — and the tester shows why that same file on the live site would block every page.
Last updated: