robots.txt Generator

Build a valid robots.txt in seconds.

User-agent group 1
robots.txt
User-agent: *
Disallow: /admin
Disallow: /private
Allow: /

Sitemap: https://example.com/sitemap.xml

About this tool

Assemble a correct robots.txt with per-user-agent allow and disallow rules, an optional crawl-delay, and your sitemap URL. Copy or download the result.

How to use

  1. Add user-agent groups (or use * for all crawlers).
  2. Add Allow and Disallow paths, and an optional crawl-delay.
  3. Add your sitemap URL, then copy or download robots.txt.

Examples

Output
User-agent: *
Disallow: /admin
Allow: /
Sitemap: https://example.com/sitemap.xml

Frequently asked questions

At the root of your domain: https://example.com/robots.txt.
No. robots.txt is a crawler hint, not access control. Sensitive pages should be protected server-side.
Contact

Let’s connect

Have a Liferay or Backend project you’d like to talk through? Fill in the form and I’ll get back to you.

Book Calendar