What is incrementality?
Definition
Incrementality assesses the true impact of an advertising campaign by isolating the outcomes definitively resulting from the marketing activity from those that would have happened without the campaign. In other words, it answers the question, “Would I have gotten those sales anyway, without running ads?” This method helps advertisers understand how much of their success can be directly attributed to a specific campaign.
“True incrementality testing requires a fully isolated holdout group with zero impressions. Without this, results are biased and can’t accurately measure incremental impact. Advertising-free holdouts ensure any uplift is due to the campaign, not external factors. This method definitively proves incremental value.” Martin Froschauer, Senior Product Manager, Measurement
The importance
Understanding incrementality is crucial for advertisers that want to understand the true value of their marketing efforts. It provides clear insights into return on investment, helping to evaluate the impact of specific channels and partners, and enables brands to prove their campaigns’ effectiveness and make informed, long-term budget decisions.
How to measure incrementality
Measuring incrementality involves designing a test that can isolate and evaluate a specific campaign’s impact on a desired outcome, such as sales or user engagement. Start with randomly assigning participants to a treatment group that receives ads and a control group that doesn’t. Then, compare the outcomes of the two groups before, during, and after the ads are delivered to determine incremental effect.
For larger clients, incrementality tests are particularly effective due to the higher data volume and complexity of their marketing mix, which often includes multiple channels.
How does incrementality differ from traditional attribution methodologies?
Marketers should consider both attribution and incrementality when measuring campaign effectiveness. Attribution is useful for short-term optimization, allowing for quick adjustments based on daily or weekly results. Incrementality, in contrast, provides long-term insights, typically reviewed annually to assess overall channel performance. Combining both methodologies offers a more comprehensive view than relying on just one.
Common attribution models:
- Post-click attribution: assigns all credit to the channel a user interacted with after a click, focusing on actions taken until conversion
- Last-click attribution: assigns all credit to the final click before conversion often overvaluing the last touchpoint and undervaluing earlier interactions
- Multi-touch attribution: spreads credit across all touchpoints, but may not fully capture the true incremental impact of each touchpoint.
While attribution focuses on assigning credit to specific touchpoints, it doesn’t confirm the added value of marketing efforts. Incrementality, however, measures the difference between a treatment group exposed to the ad and a control group not exposed, offering insights into net overall lift effectiveness.
Advantages of incrementality:
- True impact: isolates the effect of an ad to determine outcomes directly attributable to a marketing effort.
- Causal measurement: Reveals how much of an observed outcome can be directly linked to the campaign, compared to just showing correlations.
- Clear ROI calculation: offers a clearer understanding of return on investment by showing the actual contribution of each marketing activity.
- Budget optimization: Focuses on driving new conversions that wouldn’t have happened without the ad, preventing wasteful spending.
- Better decision-making: Provides precise data on what drives results, enabling better budget allocation.
- Enhanced campaign effectiveness: enables advertisers to optimize strategies by understanding the true impact of different channels and tactics.
Understanding various methodologies for incrementality testing
There are several methodologies for incrementality testing, varying in complexity and suitability for different environments.
Geo-based experiments segment users based on geographic regions to form control and treatment groups. This approach is privacy friendly as it does not rely on individual tracking.
The most common geo-based experiment is Geo-Lift Testing.
User-level experiments involve randomizing the audience at an individual level to create test and control groups. These experiments often use pseudonymized user data to protect privacy while ensuring precise measurement of ad impact on specific individuals. However, privacy regulations must be carefully adhered.
User-level experiments include:
| Intent to Treat | Public Service Announcement | Ghost Ads | Ghost Bids | |
|---|---|---|---|---|
| What is it | Basic test that splits the full audience into control and treatment groups | Serves real ads to both treatment and control groups, with control group receiving non-branded ads | When a control group user would have been shown an ad, a “ghost” ad is recorded instead of a real ad | Adapted from Ghost Ads, tracking control group bids and recording “ghost” impressions that don’t result in actual ad placements, tracking potential exposure |
| When to use it | Across open and closed environments when no cross-site user IDs available | Open environments with limited data volume; good for faster statistical significance at higher costs | Works in closed environments where auction process is fully visible, e.g. owned inventory like social channels | Open environments, offering highest level of precision and scalability across acquisition, retention and retargeting |
| Pros | Simple setup; not cookie dependent | More reliable than ITT, creating a more controlled test environment with both groups seeing ads | Avoids costs for non-branded ads; can effectively compare exposed users with those who would have been exposed | Most accurate method for open web advertising; significantly reducing noise; scalable across the funnel |
| Cons | Noisier due to some treatment group users not seeing ads | Costs to serve non-branded PSAs; results may be biased | Limited to closed environments and cannot be used on the open web; cookie dependent | Complex setup; cookie dependent |
Maximize incrementality testing on the open web
Ghost Bidding is the most advanced and rigorous methodology available for open web advertising. Ghost Bidding is adapted from ghost ads approach used in closed environments but is designed for the open web.
In ghost ads, a false bid is placed that never actually produces an ad. The advertiser then compares the treatment group with users who would have seen an ad if the bid had been real. However, Ghost Ads can only be used in closed environments where the entire auction process is visible.
Ghost Bidding takes a similar approach, but is optimized for the open web. A bid is placed as usual for the treatment group, while the control group is tracked to determine if the bid would have won the auction. If so, a “ghost impression” is recorded for the control group user. This allows for an accurate comparison of exposed users to those who would have been exposed, without the limitations of ghost ads in an open environment.
At Criteo, we leverage a Ghost Bidding approach using a random split where only users with a high probability of being reached are analyzed.
Our methodology offers several benefits:
- Significantly reduces noise for more accurate results by focusing only on users with a high probability of being reached
- Can be applied with an always-on approach across acquisition, retention, and retargeting tactics focused on conversion
- Like Ghost Ads, maintains a true control group of users that are not exposed to any ads
Getting started
Incrementality is a vital measurement method for understanding the true value of your advertising campaigns. Carefully selecting the right methodology and ensuring proper test setup will help you generate actionable insights, optimized strategies, and more informed budget decisions.
To enhance robustness and impartiality of your incrementality testing, consider collaborating with third-party experts like Measured, Nielsen and Rockerbox. “We support and encourage partners to test incrementality with third-party measurement parties who bring deep expertise, industry best practices and unbiased perspectives.” Martin Froschauer, Senior Product Manager, Measurement.
Get started today and talk to an expert.
For further resources on incrementality, explore our success stories with French retailer, Gémo, and Publicis Groupe’s premier integrated agency, Digitas.
