Robots.txt and sitemap checker
Enter a site to read its robots.txt, test the root path against 11 crawlers, and check the XML sitemap and llms.txt.
Free. No account. 15 seconds. We read only the public page.
What you get
- Allowed or blocked status for each crawler at the site root.
- Sitemap lines, whether the sitemap loads, URL count and lastmod coverage.
- llms.txt status.
- The first part of your robots.txt text.
How rules are evaluated
A crawler uses the group that names it, or the * group if none does. Within a group the longest matching rule wins, and Allow wins ties. We test the root path / only, so path-specific rules deeper in the site are not evaluated.
Frequently asked questions
Does a missing robots.txt hurt?
No. Crawlers assume everything is allowed. A robots.txt is still useful for declaring your sitemap.
Can I block AI training but stay in AI search?
Yes, by disallowing training bots such as GPTBot while allowing search bots such as OAI-SearchBot.
Related guides and tools
Want the issues fixed for you?
RankFlow Systems handles the fixes and the content.