Robots.txt Checker
Enter a website URL to retrieve its robots.txt file and review the crawl instructions served for that host.
Results will appear here after the check runs.
The file is retrieved by the TipsMake server. Crawler support, cached copies, host variations, and rendered site behavior can affect real-world crawling.
How to Use Robots.txt Checker
- Enter the site's absolute homepage or robots.txt URL.
- Run the check and review the HTTP response, user-agent groups, Allow and Disallow paths, and Sitemap lines.
- Test important URL paths against the intended crawler rules, publish corrections at the host root, and check again.
What Robots.txt Can and Cannot Do
A robots.txt file controls crawler access by user-agent and path pattern. It belongs at `/robots.txt` on each protocol and host it governs. Rules are case-sensitive for paths, and the most specific matching rule can determine access for supported crawlers.
Blocking crawling does not guarantee that a known URL disappears from search, and robots.txt is not an access-control system. Sensitive content requires authentication or authorization, while removal from search may require an indexable noindex response before crawling is blocked.
Robots.txt Checker Example
With `User-agent: *` and `Disallow: /private/`, a supported crawler is asked not to crawl https://example.com/private/report. `Sitemap: https://example.com/sitemap.xml` advertises the sitemap location.
Frequently Asked Questions About Robots.txt Checker
Where must robots.txt be located?
It must be available at the root of the protocol and host it controls, such as https://example.com/robots.txt.
Does Disallow remove a page from search results?
Not necessarily. It prevents supported crawlers from fetching the path, but a URL discovered elsewhere can still be known or displayed without a crawled snippet.
Can robots.txt protect private files?
No. It is a public set of crawler instructions, not authentication. Protect private resources with server-side access controls.