Robots.txt Generator

Control how search engines crawl your website

Quick Presets

User Agents (Bots)

*
Common bots:

Disallow Paths

/admin/
/private/
Common paths:

Allow Paths (Exceptions)

No specific allow rules

Additional Settings

Leave empty to not set a crawl delay

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

How to use:

  1. Download the robots.txt file
  2. Upload it to your website's root directory
  3. Access at: yoursite.com/robots.txt

Important Notes

  • robots.txt is a suggestion, not enforcement
  • Malicious bots may ignore robots.txt
  • Use "/" to block everything
  • Test with Google Search Console

Frequently Asked Questions

What is a robots.txt file and what does it do?

A robots.txt file is a plain text file placed in your website's root directory that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It is the first file crawlers check before indexing your site.

Where should I place the robots.txt file on my website?

The robots.txt file must be placed in the root directory of your domain, accessible at yourdomain.com/robots.txt. Search engines will only look for it at this exact location, so placing it in a subdirectory will have no effect.

Can robots.txt block AI crawlers like ChatGPT and Google AI?

Yes, you can add specific user-agent rules to block AI crawlers such as GPTBot, ChatGPT-User, Google-Extended, and others. Our generator includes presets for common AI crawlers, making it easy to restrict AI training bots from accessing your content.

What is crawl delay and should I set one?

Crawl delay tells bots to wait a specified number of seconds between requests to your server. It is useful for smaller servers that may be overloaded by aggressive crawling, but major search engines like Google ignore this directive and use their own throttling.

Should I include my sitemap URL in robots.txt?

Yes, adding a sitemap directive in your robots.txt file helps search engines discover your XML sitemap faster. This improves crawl efficiency by giving search engines a complete map of all the pages you want indexed on your site.