Robots.txt Generator

Control how search engines crawl your website

Quick Presets

User Agents (Bots)

*
Common bots:

Disallow Paths

/admin/
/private/
Common paths:

Allow Paths (Exceptions)

No specific allow rules

Additional Settings

Leave empty to not set a crawl delay

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

How to use:

  1. Download the robots.txt file
  2. Upload it to your website's root directory
  3. Access at: yoursite.com/robots.txt

Important Notes

  • robots.txt is a suggestion, not enforcement
  • Malicious bots may ignore robots.txt
  • Use "/" to block everything
  • Test with Google Search Console