What is an AI crawler?
Automated clients discover or collect web content. Search and training can have separate agents; a user-requested fetch is a different action again.
OpenAI crawler documentation ↗Know what your website allows.
Separate search discovery, training choices and the rules behind each result.
Read the policy.
Understand the difference.
Use the current rules. No additional website request.
Select a crawler to inspect its group, rule and official documentation.
Declarations only. Sitemap contents and reachability are not checked.
This report analyzes robots.txt policy. It does not guarantee access through your CDN, firewall, bot protection or authentication layer. It does not check page-level indexing directives or guarantee AI citations.
Search visibility and training permission
are different decisions.
Review before publishing. Existing robots.txt rules affect how additions behave. Specific groups override wildcard protections: merge existing private-path exclusions into any new group. Do not replace your full file blindly.
Select a policy, then generate additions.
Google-Extended combines specified training and grounding uses. User-triggered fetchers with special behavior are excluded from generated access controls.
Automated clients discover or collect web content. Search and training can have separate agents; a user-requested fetch is a different action again.
OpenAI crawler documentation ↗It declares crawler preferences for an origin. It is not an access lock or an indexing guarantee. More specific paths win; equivalent Allow rules take precedence.
Robots Exclusion Protocol ↗Training restrictions are not automatically search restrictions. Google-Extended and Applebot-Extended control uses of collected content rather than separate crawling agents.
Google’s policy-token guidance ↗Provider behavior can change. Every result links to its official source.
A few useful answers about crawlers, permissions and your report.
It reads robots.txt and evaluates declared rules for the listed tokens. It does not impersonate providers or test their real network access. Firewalls, CDNs, authentication and rate limits can still prevent retrieval.
Not automatically. GPTBot and ClaudeBot have training purposes distinct from their providers’ search crawlers. Choose training permissions according to your publishing policy; review search access separately.
Review the OAI-SearchBot group and the public paths you want discoverable. Permit the intended paths while preserving private-path restrictions. Permission does not guarantee a citation, ranking or inclusion.
Claude-SearchBot supports search, Claude-User retrieves pages for user requests, and ClaudeBot collects material that may contribute to training. Anthropic documents robots.txt controls for all three.
No. It is a policy token for specified Gemini training and grounding uses through existing Google crawlers. It is not a separate HTTP crawler and does not control inclusion or ranking in Google Search.
It controls whether Applebot-collected content may be used for generative-model training. Applebot itself supports search discovery. Restricting the Extended token does not itself block Applebot crawling.
Provider behavior differs. OpenAI says robots.txt may not apply to ChatGPT-User actions; Perplexity says Perplexity-User generally ignores it. The tool shows declared rules without presenting them as enforceable access guarantees.
HTTP 404 is shown as no robots restriction declared. A timeout, denied request, server failure or invalid response produces an unknown policy result. Neither outcome proves that a provider can reach your content.
No. They are reviewable additions only. Specific agent groups override wildcard groups, so adding one can change existing protections. Merge carefully, preserve account and checkout exclusions, then test important paths.
Website responses are temporarily cached for 10 minutes to avoid repeated requests. Hashed rate-limit keys are short-lived. This tool creates no saved reports, and pasted rules stay in your browser. No email or login is required.
Connect technical SEO, content structure and your brand experience.