LonaDev · Free SEO utilities

Robots.txt Generator

Create a robots.txt file with user-agent, allow, disallow, optional crawl-delay and sitemap directives. Review rules before publishing.

Free to useNo account requiredMobile friendlyInstant results

Apply rules to all crawlers (User-agent: *)Allow paths (one per line)Example: /public/Disallow paths (one per line)Example: /admin/ or /private/Sitemap URLCrawl-delay (optional seconds)

Important: robots.txt is a crawl directive, not access control. Disallowed URLs may still be indexed if discovered elsewhere. Protect private content with authentication and server-side authorization. Google does not support Crawl-delay.

robots.txt preview

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

More free website tools

SEO checker Speed test Meta description generator Sitemap generator Robots.txt generator

How this robots txt generator works

Generate robots.txt rules for search crawlers. Set user agents, allowed and blocked paths, and your sitemap URL, then download the file.

Frequently asked questions

Where should robots.txt be located?

For a host, it is normally published at the root path, such as https://example.com/robots.txt. Rules apply to that protocol, host and port.

Does robots.txt protect private pages?

No. It controls crawler access but is not authentication. Use server-side access controls to protect private or sensitive content.