# Did the Advertising Create More Sales? Plan an Incrementality Test

Plan an incrementality test around a real spending decision. Choose a credible comparison, verify delivery and judge uncertain results against contribution.

Source: https://dtcup.com/blogs/insights/incrementality-testing-for-acquisition-a-practical-audit/

DTCUP / FIELD NOTES / OPERATE

# Did the Advertising Create More Sales? Plan an Incrementality Test

Plan an incrementality test around a real spending decision. Choose a credible comparison, verify delivery and judge uncertain results against contribution.

Published Apr 11, 2026Updated Sep 20, 202611 min read

 ![Jack Hu](https://dtcup.com/wp-content/plugins/dtcup-knowledge/assets/jack-hu-small.jpg)**Jack Hu**7 years of experience building and operating brand websites, specializing in website development, paid advertising and SEO. Built a brand website from launch to US$10 million in annual sales.



  On this page  ## The Short Answer

Specify the advertising change, eligible population, comparison, outcome and business decision before choosing a study. Assess whether the available data and budget can answer the economically relevant question, and document the analysis and operational limits before launch.

Verify actual assignment, campaign delivery and outcomes in both conditions. Preserve the comparison, account for delayed results and report uncertainty with the estimate. Judge incremental contribution as well as revenue, then apply the finding only within the tested scope rather than treating attributed sales or a before-and-after chart as a causal answer.

## Key takeaways

- A test of an existing campaign and a test of more budget answer different decisions.
- Choose a feasible comparison and analysis before committing spend.
- Verify the intended delivery difference and measure outcomes in both conditions.
- Read uncertainty and contribution before applying the result beyond its tested scope.

An advertising report can tell you which purchases received credit under its rules. It does not directly tell you how many of those customers would have bought anyway. That distinction matters when deciding whether to keep spending, add budget or try a different channel.

An incrementality test tries to answer a specific version of the second question. It compares outcomes under a defined advertising change with a credible estimate of what would have happened without that change. A holdout is a group deliberately kept under the comparison condition so the business has something more useful than last week's sales to judge against.

I would begin with the spending decision, not the testing tool. A study that evaluates an entire campaign is not automatically an answer about the next dollar of budget. Write down what will change, what will remain in place and what decision the result could support before making any account changes.

## Define the advertising change and the business decision

There are several useful questions, but they are different tests. You might ask whether introducing a new activity adds sales, what happens when an existing activity is removed, or whether increasing current spending produces enough extra contribution. Define which comparison you actually need.

For a fictional brand, the question could be: “Should we expand this new acquisition campaign if the additional orders it creates can cover its additional cost?” The comparison is the new campaign versus the existing marketing arrangement without it. Other advertising may continue in both conditions. This is not a test of all advertising against no marketing at all.

Write a short study brief:

| Part of the brief | What the team needs to agree |
|---|---|
| Decision | Keep, stop or expand a specific advertising activity |
| Eligible population | Customers or markets to which the conclusion should apply |
| Treatment and comparison | The exact difference in activity between the groups |
| Main outcome | A defined purchase, qualified inquiry or other relevant result |
| Commercial boundary | Costs and order contribution needed for the decision |
| Timing and limits | Study period, later outcome observation, spending limit and operational stop conditions |

Do not let the wording change after the result arrives. If the study measured the effect of removing one campaign, describe that result in the context of the remaining marketing. It does not establish what the campaign would achieve alone in a new country.

Agree who can approve the study, implement it, inspect delivery and make the final commercial decision. A test can intentionally change revenue and spending. The account operator needs a specific authorized plan, including what happens when the study ends.

## Choose a comparison the business can actually maintain

A supported user-level lift study can create treatment and comparison groups through the advertising platform's experiment system. Check availability, campaign coverage, measurement requirements and what is withheld. Google's [Conversion Lift documentation](https://support.google.com/google-ads/answer/12003020?hl=en) describes controlled lift studies and makes clear that access is not available to every account. Do not assume that duplicating a campaign creates the same experiment.

A geographic study changes advertising across defined regions. This can be useful when the decision is better evaluated at market level or when individual-level assignment is impractical. However, choosing one strong city and one weak city does not create a fair comparison merely because both have residents.

The [Google Research overview of matched-market design](https://research.google/pubs/a-time-based-regression-matched-markets-approach-for-designing-geo-experiments/) explains why a small number of regions, different market sizes and practical constraints complicate assignment. A modeled comparison also depends on its assumptions. Have the person designing the study explain those assumptions and how the available history supports them.

Use historical outcomes to examine whether the proposed design is plausible before launch. Look at the eligible population, sales pattern, unusual market events and whether the advertising can be delivered according to the plan. Do not select or discard markets after seeing which arrangement produces the most attractive lift.

Avoid an uncontrolled before-and-after comparison when you need a causal answer. Sales may change because of seasonality, availability, pricing or other marketing. Such a comparison can describe what happened and suggest an investigation, but it usually leaves too many competing explanations to establish what advertising caused.

The important question is not which method sounds most sophisticated. It is whether the business can create and observe the intended difference with a comparison that supports the decision. If it cannot, record that limitation instead of giving an ordinary trend chart an experimental label.

## Check feasibility before committing the budget

Ask what size of improvement would change the business decision and whether the proposed study could distinguish that improvement from ordinary variation. A test that can only detect an enormous effect may not help decide between two economically similar options.

Use the business's actual history and the chosen method to plan this assessment. Outcome frequency, variation, market structure, assignment and the available budget all matter. There is no universal number of orders or a universal two-week duration that makes every test reliable.

The [research summary on randomized paired geo experiments](https://research.google/pubs/trimmed-match-design-for-randomized-paired-geo-experiments/) identifies a small number of regions, unequal outcome distributions and changes over time as design difficulties. Those are reasons to inspect the available data and uncertainty, rather than treating the number of rows in an export as sufficient evidence.

Get an analysis plan before launch. It should identify the main outcome, comparison method, uncertainty reporting, treatment of delayed outcomes, relevant exclusions and the scheduled decision point. If the design needs specialist statistical work, commission that while you can still change the plan, not after spending has finished.

Include the operational cost of learning. A holdout can mean withholding an activity that may help sales, while a budget-increase test requires extra spending. The business should understand that exposure and have limits it can afford. A technically interesting study is not automatically the best use of a small brand's current resources.

If the feasible study is too uncertain, improve measurement, simplify the question or postpone the experiment. Continue making bounded operating decisions from the evidence available, with its limitations visible. “We cannot currently measure this precisely” is more useful than pretending the activity has no effect or definitely works.

## Verify assignment, delivery and outcome measurement

Keep the assigned groups stable according to the study design. Do not redefine the treatment group as only the people who later clicked or actually noticed an ad; that selects people based on behavior after assignment and changes the comparison. Use the method's prescribed analysis population.

Measure the intended outcome in both conditions. If the business question concerns all qualifying orders, counting only orders carrying an ad click identifier would miss the very distinction the study is trying to examine. Confirm that the data source, status rules, currency and timing are suitable for the design.

Use the [event-definition guide](https://dtcup.com/blogs/insights/ecommerce-event-taxonomy-a-practical-audit/) to establish the purchase trigger and relevant identifiers. Check duplicate senders, test orders, cancellations and refunds as applicable. Measurement must respect the customer's privacy choices; a cleaner experiment does not justify ignoring them.

For a geographic study, inspect all campaigns that could reach the study markets. Check that another campaign is not unintentionally replacing the withheld activity or sending the same promotion into both conditions. Consider movement across regions and media that do not respect the proposed boundary. Record remaining exposure limitations.

Budget behavior is part of implementation. Removing regions from a constrained campaign can change where the remaining budget is spent, which may alter the comparison. Google's [geo-experiment implementation guidance](https://support.google.com/google-ads/answer/18073432?hl=en) specifically addresses targeting, campaign budgets and separation between study conditions. Follow the current instructions for the actual campaign setup rather than assuming an exclusion alone preserves the baseline.

Before starting, save the approved settings and confirm the changes in the account. After starting, inspect actual delivery and spending against the plan. A configuration screenshot is evidence of a setting, not proof that the intended advertising difference occurred throughout the study.

## Protect the comparison while the study runs

Monitor whether the study is operating correctly: the right campaigns, markets, spending, tracking and customer journey. Also record events that can change interpretation, such as stock shortages, a market-specific promotion, a delivery disruption or a website outage.

Decide in advance how the analysis will handle serious disruptions. Do not remove an inconvenient day simply because it makes the result look worse. If an unplanned problem makes the original analysis unsuitable, document the reason and the revised limits. Sometimes the honest outcome is that the study could not answer the intended question.

Keep routine optimization from accidentally erasing the treatment. If someone changes creative, targeting or budgets during the test, record what changed and whether it remains within the design. A major redesign halfway through does not produce a clean result for the original plan.

Use the planned review point and statistical method. Repeatedly ending a study whenever the result looks favorable can undermine the interpretation of a fixed-duration analysis. Operational stop conditions are different: a serious customer problem or breach of the agreed spend limit may justify stopping, but the resulting evidence needs to be labeled accordingly.

Allow the specified outcome observation period. Orders can occur after the advertising exposure, and later cancellations can change the commercial result. Do not quietly replace the chosen window with whichever cutoff makes the result most persuasive.

At completion, restore or change the campaigns under the agreed plan and verify their state. The study's end is not permission to leave a temporary market exclusion or increased budget running indefinitely.

## Read the result as an estimate, not a magic number

Consider a simplified fictional example with equally sized, properly assigned groups of 10,000 eligible people. Suppose the defined observation produces 460 qualifying orders in treatment and 400 in comparison. The simple observed difference is 60 orders, and the relative difference against the comparison is 15%.

That arithmetic explains the idea; it is not a complete statistical analysis or evidence from an actual campaign. Unequal group sizes, geographic designs and other methods need the appropriate estimator. A real report must include the method's uncertainty and checks on whether the study operated as intended.

Now suppose the relevant additional advertising cost was $6,000 and the estimated additional order revenue was $9,600. Under these invented assumptions:

| Calculation | Illustrative result |
|---|---|
| Estimated additional orders | 460 − 400 = 60 |
| Additional cost per estimated additional order | $6,000 ÷ 60 = $100 |
| Estimated incremental revenue per advertising dollar | $9,600 ÷ $6,000 = 1.6 |
| Contribution before advertising, at an assumed 40% on that revenue | $9,600 × 40% = $3,840 |
| Contribution after the additional advertising cost | $3,840 − $6,000 = −$2,160 |

This example shows why positive additional revenue is not enough to establish profitability. The assumed contribution rate must reflect the appropriate costs and sales mix, and any further costs or later customer value need their own justified treatment. Do not insert an optimistic lifetime value simply to turn a weak result positive.

Read the uncertainty alongside the point estimate. A range spanning both an unattractive and an attractive commercial outcome may leave the budget decision unresolved even when the central estimate looks promising. A result that cannot clearly distinguish an effect from zero does not prove the true effect is exactly zero.

Likewise, dividing by a very uncertain or near-zero estimate of additional orders can produce an unstable cost figure. Do not present a precise-looking ratio while hiding that uncertainty. Ask the analyst to explain what the result supports about the actual decision, including what it cannot rule out.

## Apply the learning within the tested scope

Write the conclusion with the audience, markets, dates, campaign setup, spend contrast, outcome definition and limitations. Include the commercial implication and the next action. “The test supports keeping this activity at the tested scope” is more useful than declaring an entire channel universally effective.

A test of higher spending evaluates the particular increase under those conditions. It does not demonstrate that every further increase will return the same amount. Similarly, evidence from a mature market may not carry over to a market with different distribution, customer knowledge or product availability.

Do not add results from several isolated channel tests as though they partition all revenue. Marketing activities can interact, and each study describes its own comparison with the other activity that remained in place. Keep experiment evidence distinct from platform attribution. The [attribution reconciliation guide](https://dtcup.com/blogs/insights/attribution-reconciliation-a-practical-audit/) explains why credited sales are a different view.

Make a proportionate decision: maintain the activity, adjust it, stop it, or improve the design and investigate again. If the result is inconclusive, explain whether the main problem was measurement, execution, insufficient information or a commercially small difference. Each leads to a different next step.

Preserve the brief, settings, changes, source data definitions and analysis. This lets the next team understand what was learned without rebuilding the study from a screenshot. The [analytics guides](https://dtcup.com/guides/analytics/) connect this work to reliable events, commercial reporting and decision ownership.

## Questions about incrementality testing

### Does a high reported ROAS mean we do not need a holdout?

No. Reported ROAS describes revenue credited under the platform's attribution rules. A holdout addresses what changed because of the tested activity. Whether a study is worth running depends on the decision, expected value of better information and whether a credible experiment is feasible.

### Can we compare two cities and call the sales difference incremental?

Not without a defensible design. Cities can differ in underlying demand, history, promotions and availability. Review assignment or modeling assumptions, historical outcomes, delivery and uncertainty with a qualified analyst. A pair of totals alone does not show what would have happened without the advertising change.

### What should we do if the test is inconclusive?

Do not relabel it as proof of no effect or as a positive result. Check execution and measurement first, then whether the design could answer the economic question. Keep the uncertainty visible while making a bounded business decision, and only commission another study when its design is likely to provide more useful evidence.



 ## Sources

1. [support.google.com — 12003020](https://support.google.com/google-ads/answer/12003020?hl=en)
2. [research.google — a-time-based-regression-matched-markets-approach-for-designing-geo-experiments](https://research.google/pubs/a-time-based-regression-matched-markets-approach-for-designing-geo-experiments/)
3. [research.google — trimmed-match-design-for-randomized-paired-geo-experiments](https://research.google/pubs/trimmed-match-design-for-randomized-paired-geo-experiments/)
4. [support.google.com — 18073432](https://support.google.com/google-ads/answer/18073432?hl=en)

CONTINUE / THE TOPIC

## Keep the next question connected.

[NEXT BEST READ**Design Website Measurement Around the Decisions You Need to Make**](https://dtcup.com/blogs/insights/ecommerce-event-taxonomy-a-practical-audit/)[RELATED GUIDE**Why Your Advertising and Store Reports Disagree, and How to Investigate**](https://dtcup.com/blogs/insights/attribution-reconciliation-a-practical-audit/)[RELATED GUIDE**Work Out Whether Your DTC Business Can Make Money Before You Scale It**](https://dtcup.com/blogs/insights/contribution-margin-systems-a-practical-audit/)[RELATED GUIDE**Diagnose Duplicate Purchase Events Without Losing Real Orders**](https://dtcup.com/blogs/insights/diagnose-duplicate-purchase-events/)

Still have a question?[Ask the DTCUP Community ↗](https://dtcup.com/community/)

 



  FAQ

## Questions related to this insight.



Clear scope, practical constraints and direct answers for the decision ahead.



Does a high reported ROAS mean we do not need a holdout?No. Reported ROAS describes revenue credited under the platform's attribution rules. A holdout addresses what changed because of the tested activity. Whether a study is worth running depends on the decision, expected value of better information and whether a credible experiment is feasible.



Can we compare two cities and call the sales difference incremental?Not without a defensible design. Cities can differ in underlying demand, history, promotions and availability. Review assignment or modeling assumptions, historical outcomes, delivery and uncertainty with a qualified analyst. A pair of totals alone does not show what would have happened without the advertising change.



What should we do if the test is inconclusive?Do not relabel it as proof of no effect or as a positive result. Check execution and measurement first, then whether the design could answer the economic question. Keep the uncertainty visible while making a bounded business decision, and only commission another study when its design is likely to provide more useful evidence.
