# Separate Bots from Website Traffic Without Inventing a Human Count

Separate verified bots, suspected automation and unclassified activity. Build useful page and channel reports without presenting event counts as verified people.

Source: https://dtcup.com/blogs/insights/separate-bots-from-website-traffic/

DTCUP / FIELD NOTES / OPERATE

# Separate Bots from Website Traffic Without Inventing a Human Count

Separate verified bots, suspected automation and unclassified activity. Build useful page and channel reports without presenting event counts as verified people.

Published Aug 30, 2026Updated Sep 21, 202610 min read

 ![Jack Hu](https://dtcup.com/wp-content/plugins/dtcup-knowledge/assets/jack-hu-small.jpg)**Jack Hu**7 years of experience building and operating brand websites, specializing in website development, paid advertising and SEO. Built a brand website from launch to US$10 million in annual sales.



  On this page  ## The Short Answer

Define whether you are counting requests, page views, sessions or orders before applying a bot label. Keep verified crawler identity, controlled internal automation, suspected automation and unclassified activity separate. Browser activity without a flag is not a verified human count.

Show the data source, classification coverage, exclusions and rule changes beside each report. Verify crawler claims using the provider's current method, test false positives and keep channel labels separate from identity or attribution claims. Reporting filters and security blocking need separate decisions.

## Key takeaways

- Requests, page views, sessions and people are different units.
- A user-agent name alone does not verify a crawler's identity.
- Keep uncertain classifications visible instead of silently calling them human.
- Test rule changes and explain their effect on page and channel comparisons.

A sudden increase in visits can make a website look healthier while telling you very little about customer demand. Some activity may come from search crawlers, monitoring tools, automated browsers or unwanted requests. Meanwhile, genuine visitors can be missing from a browser analytics report.

The useful goal is not a dashboard that labels everything with certainty. It is a report that separates the evidence you have, shows what remains unclassified and supports a sensible decision. A known search crawler should not inflate your customer funnel, but an unfamiliar visitor should not disappear merely because their behavior looks unusual.

I would build two connected views: an operational view of requests reaching the website, and a customer-behavior view of the activity the measurement setup can observe. Keep their definitions visible. Neither is a census of every person who considered your brand.

## Start by naming what the report counts

A server request, page view, session, user and order are different units. Loading one page can involve requests for the document, images, scripts and other resources. Repeated requests do not necessarily mean repeated visits, and a session identifier does not establish a unique human being.

Write a short definition beside every headline. For example, “document requests recorded at the delivery layer” and “page-view events accepted by the website analytics system” explain two different observations. A title such as “real visitors” promises more than either measurement establishes.

Document the site and period covered, timezone, exclusions, sampling or aggregation, and when the data was last updated. Check whether the report includes the public website, account pages, API calls, previews or staff testing. Those choices can change the apparent trend before bot detection does anything.

Keep the business question in view. If the team wants to investigate infrastructure load, resource requests matter. If it wants to understand a buying journey, relevant page and business events matter. If it wants to reconcile purchases, use the order records rather than estimating sales from a traffic label.

The [event planning guide](https://dtcup.com/blogs/insights/ecommerce-event-taxonomy-a-practical-audit/) helps establish those definitions before the team adds another classification to an already ambiguous report.

## Understand what each collection layer can see

An edge or hosting record can describe requests it handled. A browser measurement tool describes events that reached it under the implemented collection rules. An application records its own operations. Coverage can differ because of caching, request routing, browser behavior, consent, filters and failures.

Put these views beside one another without expecting equal totals:

| Source | Useful question | Important limitation |
|---|---|---|
| Delivery-layer or server records | What requests reached this part of the website? | Requests include more than customer page visits, and another layer may serve some activity |
| Browser analytics | What measured journeys and events were recorded? | The observed set is not every visitor, and browser execution does not prove humanity |
| Application records | What account, form or business operations occurred? | An operation's existence does not explain every preceding visit or its cause |
| Verified order records | Which purchases reached the defined business state? | Orders do not establish the total number of human visitors |

Google's [known bot-traffic exclusion documentation](https://support.google.com/analytics/answer/9888366?hl=en) says Analytics automatically excludes known bots and spiders. It also says you cannot disable that exclusion or see how much known bot traffic was excluded. Do not present a GA report as a complete bot inventory or claim that its remaining traffic is all human.

For another measurement product, check its own actual behavior. A similar-looking dashboard does not establish that it applies the same lists, rules or collection boundaries. Record what is unavailable as unavailable rather than reconstructing it from an unsupported subtraction.

## Use labels that match the strength of the evidence

Separate identity verification from behavioral inference. A request can be verified as a particular crawler through an appropriate method. A different request may merely resemble automation. Those are not equivalent findings.

A practical reporting scheme could use the following categories, with explicit precedence so each counted event belongs to one category:

| Category | Meaning in this report | What it does not mean |
|---|---|---|
| Verified crawler or service | Identity checked using a documented, current verification method | Every action is desirable or every request is a customer visit |
| Known internal automation | A controlled test or monitor identified through the organization's agreed method | All traffic from the same network is automatically internal |
| Suspected automation | Named signals or a detection service indicate automation | A confirmed identity or proof of malicious intent |
| Browser activity without an automation flag | Collected browser activity not flagged by the chosen rules | Verified unique people |
| Unclassified | Evidence is absent, conflicting or insufficient | A category that should be silently counted as human |

These are suggested reporting definitions, not universal industry labels. Adapt them to evidence the system actually has. If your dashboard only supports two categories, add an explanation of how uncertainty is handled before using the count for a decision.

Keep the underlying signal, reason and rule version available to the responsible analyst. The public-facing business report can be simple while the classification remains inspectable. Avoid a score with no explanation of what was scored or when the method changed.

## Verify a claimed crawler before treating it as known

A user-agent name is a claim sent with a request. It is useful for investigation but should not alone establish that the request belongs to a trusted search service.

Google's [crawler verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) describes reverse DNS followed by forward verification against the original address, or matching against its published IP ranges. It distinguishes crawler and fetcher categories. Use the applicable current method and lists; do not treat every address associated with a cloud provider as a verified search crawler.

Ask the technical owner where the source address comes from. If the website sits behind a proxy, the recorded peer may be the proxy rather than the original client. Verification must use the trusted network path and its configured address handling, not an arbitrary request field a visitor can supply.

Save the method and time of verification with the classification. If a lookup fails or the required signal is missing, keep the result unconfirmed. A timeout does not establish that the visitor is fraudulent, and a familiar name does not fill the evidence gap.

For your own uptime monitor or deployment test, use a controlled identification method and a documented scope. Do not exclude a broad office or provider network if it also carries real customers. Known automation should be identifiable without teaching the report to ignore unrelated activity.

## Treat behavioral scores as clues with limits

Repeated patterns, unusually high request rates or a detection service's result can help prioritize investigation. Each clue needs context. A short visit may be a real person finding the answer immediately. A long visit may be an abandoned tab. A script can perform browser actions too.

Cloudflare's [bot-score documentation](https://developers.cloudflare.com/bots/concepts/bot-score/) distinguishes automated, likely automated, likely human, verified-bot and not-computed groupings. The term “likely” matters. Available fields and controls depend on the product and setup; do not assume every website has the same detailed score or that a missing score means a human request.

Do not create a rule such as “no purchase means bot.” That would remove the very customer hesitation a conversion review needs to understand. The same applies to excluding every visit from an unexpected country or every session with no recorded engagement. Investigate the pattern and measurement setup before changing the report.

Review a representative sample of flagged and unflagged activity, including legitimate journeys that might be misclassified. Record the evidence needed to change the classification. The goal is not to prove your first rule was right; it is to understand the mistakes the rule can make.

Avoid making access-control decisions automatically from a marketing report. Excluding a category from a chart and blocking its requests are separate actions with different consequences. A security change should have its own owner, scope, verification and recovery plan.

## Build the dashboard so totals remain explainable

Keep the total observed activity for a defined source beside its mutually exclusive classifications. Show the unclassified amount and the fraction for which detection evidence was available. Make the denominator explicit when displaying percentages.

Consider this **fictional example of browser page-view events**, all drawn from one declared collection source and period:

| Classification | Observed events |
|---|---|
| Verified or controlled automation | 1,000 |
| Suspected automation | 1,500 |
| Browser activity without an automation flag | 7,000 |
| Unclassified | 500 |
| Total | 10,000 |

The 7,000 events are not 7,000 verified people. If the team displays an analysis excluding the first two groups, 7,500 events remain, including the 500 unclassified events. Label that choice. Do not quietly rename the result “human traffic.”

Keep the original observation and classification history within the approved data-retention arrangement. When a rule changes, mark the date and explain whether prior periods were recalculated. A better classifier can change the chart without any change in customer demand.

For pages, report counts under the same source and filter basis. Separate page documents from resource requests and avoid ranking raw URLs that contain private tokens or personal information. Use an approved path-normalization rule and preserve meaningful page differences rather than stripping every parameter mechanically.

## Keep channel analysis separate from identity claims

UTM fields describe campaign labels carried in an address. They do not prove that a visit is a person, that the named campaign caused a purchase or that the label is accurate. A crawler can request a tagged link, and a customer can revisit or share one.

Apply the documented traffic classification before comparing channel behavior, but retain a view that reveals excluded and unclassified activity by source. Otherwise one channel can appear to improve merely because more of its observations were filtered out.

Keep original campaign labels and the normalized channel mapping separate. For example, a team may intentionally group email variants under one reporting channel while retaining the original source and campaign for investigation. Record changes to that mapping rather than rewriting history without explanation.

Connect observed journeys to valid business outcomes only through the approved measurement model. A verified order is stronger evidence of a transaction than a page-view event, but it does not magically identify all visitors or prove which campaign created the demand. The [attribution reconciliation guide](https://dtcup.com/blogs/insights/attribution-reconciliation-a-practical-audit/) explains why credit and actual business outcomes need different views.

If a traffic spike creates no matching customer questions, product actions or orders, investigate it. Do not immediately call it fake, and do not increase advertising spend merely because the top-line chart rose. Compare affected paths, collection changes, campaign activity and classification coverage.

## Test changes before letting them rewrite the business story

Try a revised rule against an appropriate historical sample or a separate observation view before using it in the main report. Compare which records change category and inspect the business paths that would be affected. Keep the original basis available for comparison.

Test known monitoring traffic, approved browser tests, ordinary customer-like journeys and records missing a classification signal. Use safe test data. Confirm that the report distinguishes no data from zero activity and that duplicate events do not inflate a category simply because a request was retried.

Give the review an owner and a decision. A finding might justify improving detection, fixing a duplicate event, annotating a reporting break or investigating infrastructure load. Some uncertain activity may reasonably remain uncertain.

Revisit the classification when the delivery provider, collection code, consent implementation or detection service changes. Retain enough explanation that a colleague can understand why this month's report differs from last month's. Technical detail belongs behind the relevant measure, not scattered across every business chart.

My recommendation is to prefer an honest, useful range of evidence over a precise-looking human count. You can still see which pages attract observed attention, which sources bring meaningful outcomes and where automation affects the website. You simply avoid claiming that the measurement knows more than it does.

## Questions about bots and website reporting

### Is all traffic left in Google Analytics human?

No. Google documents exclusion of known bots, not a guarantee that every remaining event represents a person. Collection gaps and different classification methods also affect comparisons. Use the report for the behavior it observes and describe its limits.

### Should we block every request classified as a bot?

No. Search crawlers, monitors and other authorized services can be useful. Identity, intent, infrastructure impact and access policy are separate questions. Make any blocking decision through the appropriate technical process rather than treating a chart filter as a security rule.

### Can we subtract bot requests from page views to calculate people?

Not when the counts use different sources or units, and even comparable event counts do not become unique people by subtraction. Define the observation, classification and remaining uncertainty. Label browser activity as such rather than presenting an unverified human total.

Explore the [analytics and measurement guides](https://dtcup.com/guides/analytics/) for connected work on event quality, reporting definitions and useful business decisions.



 ## Sources

1. [support.google.com — 9888366](https://support.google.com/analytics/answer/9888366?hl=en)
2. [developers.google.com — verify-google-requests](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
3. [developers.cloudflare.com — bot-score](https://developers.cloudflare.com/bots/concepts/bot-score/)

CONTINUE / THE TOPIC

## Keep the next question connected.

[NEXT BEST READ**Design Website Measurement Around the Decisions You Need to Make**](https://dtcup.com/blogs/insights/ecommerce-event-taxonomy-a-practical-audit/)[RELATED GUIDE**Why Your Advertising and Store Reports Disagree, and How to Investigate**](https://dtcup.com/blogs/insights/attribution-reconciliation-a-practical-audit/)[RELATED GUIDE**Diagnose Duplicate Purchase Events Without Losing Real Orders**](https://dtcup.com/blogs/insights/diagnose-duplicate-purchase-events/)[RELATED GUIDE**Build a Data Quality Queue Your Team Can Actually Resolve**](https://dtcup.com/blogs/insights/build-data-quality-exception-queue/)

Still have a question?[Ask the DTCUP Community ↗](https://dtcup.com/community/)

 



  FAQ

## Questions related to this insight.



Clear scope, practical constraints and direct answers for the decision ahead.



Is all traffic left in Google Analytics human?No. Google documents exclusion of known bots, not a guarantee that every remaining event represents a person. Collection gaps and different classification methods also affect comparisons. Use the report for the behavior it observes and describe its limits.



Should we block every request classified as a bot?No. Search crawlers, monitors and other authorized services can be useful. Identity, intent, infrastructure impact and access policy are separate questions. Make any blocking decision through the appropriate technical process rather than treating a chart filter as a security rule.



Can we subtract bot requests from page views to calculate people?Not when the counts use different sources or units, and even comparable event counts do not become unique people by subtraction. Define the observation, classification and remaining uncertainty. Label browser activity as such rather than presenting an unverified human total.







 ### Leave a Reply [Cancel reply](https://dtcup.com/blogs/insights/separate-bots-from-website-traffic/?dtsa_check=1794832877#respond)

You must be [logged in](https://dtcup.com/login/?redirect_to=https%3A%2F%2Fdtcup.com%2Fblogs%2Finsights%2Fseparate-bots-from-website-traffic%2F) to post a comment.
