In the last two months, I have had extensive discussions with some of the advertising industry’s top minds about Google’s Privacy Sandbox testing, and it’s become clear that there is still a lot of confusion out there. That’s why I wanted to take a step back and answer some of the foundational questions I’ve received about the Privacy Sandbox to make it clear what is being tested and when, and what it means for our industry.
The most-asked question I often hear is whether the Privacy Sandbox works. The bottom line is that the current tests aren’t only aimed at validating Privacy Sandbox API functionality. Rather, they are diagnostics we’re using to shape the large technical investment we’ve made as an API developer and create a strong future of robust advertising performance. These tests help us isolate specific API improvements needed from Google to ensure the Privacy Sandbox delivers the best outcomes to marketers and media owners.
What is being tested?
At its core, the industry is conducting A/B tests that compare a world with third-party cookies to a world without them, leveraging a new set of API solutions instead of the OpenRTB protocol. Of Google’s proposed APIs in the Privacy Sandbox, our testing focuses on three that significantly influence advertising: Protected Audience, Topics, and the Attribution Reporting APIs.
The goal of these APIs is to preserve much of what we’re used to in terms of digital advertising, such as one-to-one targeting, SKU-level product recommendations and measurement of performance. However, the mechanics of using these APIs are different. For example, Topics API information is passed via real-time bidding, and enables server-side auctions, while the Protected Audience API supports two auction modes: on-device and server-side in a trusted execution environment (TEE).
Here’s a concise breakdown of the objectives for each of these APIs and their potential implications for the evolution of addressable advertising:
At Criteo, we’re conducting worldwide testing across all campaign types from upper-funnel to lower-funnel performance with acquisition, retention and retargeting campaigns. We’re also testing both as a DSP and SSP, allowing us to understand the impact for advertisers and publishers together.
What groups are undergoing testing?
Conducting these tests involves extensive preparation and infrastructure evolution, as discussed in detail in my previous blog post. Like any A/B test, the first steps include recruiting and labeling test and control groups.
For the Privacy Sandbox tests, three groups of Chrome users were defined by Google at the January launch:
• Treatment group: 0.75% of traffic where third-party cookies have been removed and the Privacy Sandbox is available.
• Contextual control group: 0.25% of traffic with neither third-party cookies nor the Privacy Sandbox.
• Control group: 1% of traffic with third-party cookies enabled (business as usual).
Following test setup, we will vigilantly monitor spending across all groups, ensuring test budgets remain consistent relative to each group’s size to prevent biases in our analysis.
As mentioned, the current Privacy Sandbox test operates on less than 1% of Chrome traffic. Given this small sample size and the need to distribute it among testing partners, obtaining enough ad calls requires an investment in time. While I understand the industry’s eagerness for test results, achieving statistical significance takes time. That’s why Criteo takes a scientific approach, advocating for result sharing only when statistical significance is reached, ensuring conclusive and reliable outcomes.
What will be measured?
Having partnerships with publishers, SSPs, advertisers and retailers, we understand that the Privacy Sandbox must work for the entire ecosystem for it to work at all. To ensure the viability and fairness of all advertising stakeholders, we will be analyzing three key areas in our tests: impact to publishers, impact to advertisers and overall ecosystem health. For publisher impact, we will monitor ad CPMs and volume relative to advertiser return on ad spend (ROAS). This will help us assess if advertising through the Privacy Sandbox adequately supports both the demand and supply sides.
Aligned with the CMA’s guidance, we will also monitor various digital advertising metrics that indicate the health of the ecosystem, including revenue per impression, clicks or conversion per dollar, clicks per impression, number of bid requests, percent of planned budget spent, and latency. This comprehensive analysis not only informs the CMA’s review but also provides a holistic view of advertising ecosystem performance.
What is the testing timeline?
To provide transparency into our progress at Criteo, I want to highlight specific milestones. The period from January 4th to March 11th was dedicated to testing and iteration, as depicted in the diagram below. During this phase, we meticulously prepared for formal testing by ensuring the proper recruitment rate for our test populations and receiving accurate Privacy Sandbox labels for the Topics and Protected Audiences API tests. Simultaneously, we conducted internal testing to debug and refine our implementation in preparation for stable testing.
Starting on March 11th, we initiated our stable testing phase, scheduled to extend until May 10th. During this period, our primary focus is on gathering data from the Privacy Sandbox, analyzing its impact, and assessing performance. By June 15th we plan to deliver our final report to the UK’s Competition and Markets Authority (CMA). Subsequently, the CMA will systematically evaluate our findings, along with insights from other participating industry companies, to contribute to informed decisions regarding Google’s upcoming cookie deprecation in the fourth quarter.
Beyond the sandbox: A multifaceted approach to addressability
As we continue to innovate on a variety of addressability solutions, the Privacy Sandbox is just one area of our exploration. Marketers and media owners must venture beyond it, embracing diversity in trading currencies without complete dependence on one approach. We’ve transcended the era of sole reliance on third-party cookies for addressable engagements, now witnessing a plethora of options.
Commerce data and trading mechanisms that carry its value offer myriad opportunities for relevant advertising. To optimize addressable advertising, marketers should work with their technology partners to explore alternative approaches, including interoperable IDs such as hashed emails, as well as retail media, social platforms, and the power of contextual and AI-driven strategies. Criteo’s commerce AI feeds from ID-based signals combined with semantic analysis and content-catalog and sales inferences. By leveraging this array of methods, we can unlock a more robust and effective approach to addressable advertising.
Criteo has meticulously prepared for the deprecation of third-party cookies and anticipated further browser restrictions for years, and we are able to offer a dynamic, multi-pronged addressability strategy for our clients. Our solutions prioritize scalability and interoperability, as we envision a more open, unified and efficient ecosystem, benefiting all advertising stakeholders. This commitment positions us, and our clients, at the forefront of a future without third-party cookies, resilient and adaptable to the evolving advertising landscape.
To stay on top of our updates and insights on the future of cookieless advertising, visit our Addressability Hub.
