DTC technology × growth systems — built to compound.
DTCUP / OPEN WEB TOOLS

AI Bot &
robots.txt Checker

Know what your website allows.
Separate search discovery, training choices and the rules behind each result.

Check your website
No loginNo email gateSource-backed
01 / POLICY EXPLORER

Start with your website.

Read the policy.
Understand the difference.

We check this exact origin’s /robots.txt. HTTPS is used when no scheme is provided.

Explore a sample

02 / PUBLISH WITH INTENT

Your content. Your policy.

Search visibility and training permission
are different decisions.

SUGGESTED ADDITIONS / REVIEW REQUIRED

Make the rules explicit.

Review before publishing. Existing robots.txt rules affect how additions behave. Specific groups override wildcard protections: merge existing private-path exclusions into any new group. Do not replace your full file blindly.

Select a policy, then generate additions.

Google-Extended combines specified training and grounding uses. User-triggered fetchers with special behavior are excluded from generated access controls.

03 / READ BETWEEN THE RULES

Permission is only one layer.

What is an AI crawler?

Automated clients discover or collect web content. Search and training can have separate agents; a user-requested fetch is a different action again.

OpenAI crawler documentation ↗

What does robots.txt control?

It declares crawler preferences for an origin. It is not an access lock or an indexing guarantee. More specific paths win; equivalent Allow rules take precedence.

Robots Exclusion Protocol ↗

A deliberate publishing choice

Training restrictions are not automatically search restrictions. Google-Extended and Applebot-Extended control uses of collected content rather than separate crawling agents.

Google’s policy-token guidance ↗
REGISTRY REVIEWED 18 SEPTEMBER 2026

Provider behavior can change. Every result links to its official source.

FAQ / BEFORE YOU EDIT

Clear rules.
Better decisions.

A few useful answers about crawlers, permissions and your report.

01What does this checker actually test?

It reads robots.txt and evaluates declared rules for the listed tokens. It does not impersonate providers or test their real network access. Firewalls, CDNs, authentication and rate limits can still prevent retrieval.

02Is blocking AI training bad for SEO?

Not automatically. GPTBot and ClaudeBot have training purposes distinct from their providers’ search crawlers. Choose training permissions according to your publishing policy; review search access separately.

03How do I allow ChatGPT search crawling?

Review the OAI-SearchBot group and the public paths you want discoverable. Permit the intended paths while preserving private-path restrictions. Permission does not guarantee a citation, ranking or inclusion.

04How do the three Claude agents differ?

Claude-SearchBot supports search, Claude-User retrieves pages for user requests, and ClaudeBot collects material that may contribute to training. Anthropic documents robots.txt controls for all three.

05Does Google-Extended control Google Search?

No. It is a policy token for specified Gemini training and grounding uses through existing Google crawlers. It is not a separate HTTP crawler and does not control inclusion or ranking in Google Search.

06What does Applebot-Extended do?

It controls whether Applebot-collected content may be used for generative-model training. Applebot itself supports search discovery. Restricting the Extended token does not itself block Applebot crawling.

07Why can a user-requested fetch show “Unknown”?

Provider behavior differs. OpenAI says robots.txt may not apply to ChatGPT-User actions; Perplexity says Perplexity-User generally ignores it. The tool shows declared rules without presenting them as enforceable access guarantees.

08What if robots.txt is missing or returns an error?

HTTP 404 is shown as no robots restriction declared. A timeout, denied request, server failure or invalid response produces an unknown policy result. Neither outcome proves that a provider can reach your content.

09Will the generated rules replace my existing file?

No. They are reviewable additions only. Specific agent groups override wildcard groups, so adding one can change existing protections. Merge carefully, preserve account and checkout exclusions, then test important paths.

10Are checked websites or pasted rules saved?

Website responses are temporarily cached for 10 minutes to avoid repeated requests. Hashed rate-limit keys are short-lived. This tool creates no saved reports, and pasted rules stay in your browser. No email or login is required.

FROM POLICY TO DISCOVERY

Build a more discoverable website.

Connect technical SEO, content structure and your brand experience.

Discuss a GEO / SEO audit

Your workspace

Selected services

DTCUP / YOUR NEXT PERSPECTIVE

Stay curious.
Stay connected.

Ideas for the work behind growth—websites, brands and the apps that connect them.