robots.txt Generator

robots.txt is a small plain-text file at the root of your website that tells web crawlers — search engines and other bots — which parts of your site they may and may not fetch. It is the first thing a well-behaved crawler reads when it visits, and it works through a few simple directives: User-agent names which bot a block applies to (* means all), Disallow lists paths the bot should not crawl, Allow carves out exceptions, and a Sitemap line points crawlers to your full list of URLs so they can find everything efficiently. This generator assembles those lines correctly so you do not have to remember the exact syntax.

First, robots.txt is not security. It is a public file — anyone can read yoursite.com/robots.txt — and it is a polite request, not a lock. So never list secret URLs in it (you would be advertising them), and do not rely on Disallow to keep a page out of search results: a blocked page can still be indexed if other sites link to it. To truly keep a page private, protect it with a login; to keep it out of search, use a noindex meta tag. Second, respect for the file is voluntary. The major search engines and most reputable crawlers honour it, including the AI training bots you can block here, but a hostile scraper can simply ignore it. With those two caveats in mind, robots.txt is an essential, standard part of running a site — it guides crawlers, saves crawl budget, and points them to your sitemap.

Choose a template, add the paths to disallow and your sitemap URL, optionally block AI crawlers, and copy or download the finished robots.txt. It is generated entirely in your browser — nothing is uploaded. Round out your on-page setup with the Meta Tag Generator and the Schema / JSON-LD Generator.