robots.txt Generator
Choose crawler directives and sitemap locations to generate a robots.txt file for the root of a website host.
How to Use robots.txt Generator
- Select the user agent and add only the paths that should be allowed or disallowed for that group.
- Add absolute sitemap URLs and generate the plain-text file.
- Review rule specificity, upload the file as `/robots.txt`, and test important public and blocked paths.
Designing Crawl Directives
A concise robots.txt file is easier to audit. Group rules by user agent, use path patterns carefully, and avoid blocking CSS, JavaScript, images, or API responses required to render public pages unless that restriction is intentional.
The generated file is not a security policy and does not issue noindex directives. Confirm behavior with crawler testing tools and make sure the same host and protocol serve the final file without an unexpected redirect or error.
robots.txt Generator Example
To discourage crawling of an internal search area while advertising the main sitemap, use `User-agent: *`, `Disallow: /search/`, and `Sitemap: https://example.com/sitemap.xml`.
Frequently Asked Questions About robots.txt Generator
Should I block admin and login URLs in robots.txt?
You may reduce crawler requests, but the public file reveals those paths and does not secure them. Authentication and authorization remain necessary.
Can I add noindex to robots.txt?
No. Major search engines do not support robots.txt as a reliable noindex mechanism. Use a robots meta tag or X-Robots-Tag on a crawlable response.
Can one robots.txt file control subdomains?
No. Each host uses its own root robots.txt file, so www.example.com and shop.example.com need separate files when both are used.