Robots.txt Generator

🤖 Agent Ready

Generate a robots.txt file with user agents, allow/disallow rules, crawl delay, and sitemap

Rule Group 1
User Agent
Disallow Paths
Allow Paths

No rules — click Add

Crawl Delay (seconds)
Sitemap URL
Generated robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

How to use this robots.txt generator

A robots.txt file tells search engine crawlers which paths they can or cannot crawl. Use this generator to create valid rules for User-agent, Allow, Disallow, optional Crawl-delay, and your sitemap URL.

Start with one group for *, then add bot-specific groups only when needed. Keep rules minimal and test them in Google Search Console before deploying.

Robots.txt examples

Block admin pages, allow everything else

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Allow one folder for Googlebot while blocking others

User-agent: Googlebot
Allow: /blog/
Disallow: /

User-agent: *
Disallow:

Common robots.txt mistakes

  • Blocking important pages by accident: double-check Disallow: / rules before publish.
  • Using robots.txt to hide sensitive content: robots.txt is public, so it is not a security control.
  • Forgetting sitemap links: add a sitemap so crawlers discover new pages faster.
  • Confusing crawl vs index: if you need de-indexing, use meta robots or headers too.

Related Tools

Meta Tag GeneratorHTTP Status CheckerXML FormatterURL Encoder