The start of 2024 marked a milestone in the deprecation of third-party cookies, with Google beginning to block them on 1% of Chrome traffic. That moment officially kicked off a five-month testing period to compare advertising performance across audiences who are still served ads with cookies and those whose advertising is supported with API tools from Google’s Privacy Sandbox. These tests involving 1% of Chrome’s traffic will continue to run through May and serve as the basis for the UK’s Competition and Markets Authority (CMA) evaluation to determine the next steps in the potential 100% deprecation of third-party cookies later this year.
These tests are crucial for the future of the advertising ecosystem, but lately, I’ve noticed a lot of industry calls placing ownership and responsibility on advertisers rather than the tech providers that serve them. While marketers are no strangers to testing, from A/B to conversion optimization, asking them to take on testing Privacy Sandbox APIs would require large efforts on their part to not only execute the tests, but to ensure proper analysis and statistical significance of their results. And it’s fair to say that most advertisers I speak with simply don’t have the right resources to do this with an entirely new system. That’s why the burden of testing should rather be shouldered by tech partners who have deep tech knowledge and experience that makes them better equipped to implement Privacy Sandbox APIs and verify accurate and meaningful results.
It’s also important to note that testing of the Privacy Sandbox should not be done in a vacuum, but in ways that can be compared and contrasted to addressability alternatives. I’ll comment on the specifics of that topic in a later post, but for now, let’s look solely at the testing approach we’re taking with Privacy Sandbox.
Testing timeline and metrics
The timeline below highlights Criteo’s main testing stages over the next six months. We’re currently ahead of the iteration period, having begun testing in early January, and we will continue evaluating all testing mechanisms to confirm they’re working properly and that we’re receiving the correct metrics. We will then move into a two-month locked testing period in mid-March, which will serve as the basis for our final data report to the CMA that will be delivered by June 15th at the latest. As we’ve done before, we will keep you appraised of what we learn as we progress towards making that report.
About our testing approach and methodology: We have a duty to our clients, partners and the wider industry to ensure the economics of advertising continue to work for everyone. That’s why the global metrics we will analyze, not only for the CMA, but for the interest of our wider industry, are geared toward advertiser performance. Our testing will largely focus on constant return on ad spend and how this impacts publisher CPMs. For advertisers, maintaining positive ROAS is crucial, and healthy publisher CPMs depend on ROAS. If ROAS decreases, CPMs will follow as marketers will likely reduce their budgets or reallocate investments to other channels which provide predictable performance. Further metrics we will be fully evaluating include volume metrics, revenue, clicks and conversions per dollar, clicks per impression, latency and number of bid requests received.
Navigating the testing terrain
Testing the new APIs isn’t like setting up standard tech integrations—it’s incredibly intricate and requires significant engineering investment. At Criteo, we have 1,000 product and R&D employees, many of whom have been dedicated to crafting and evaluating solutions for the future of addressability, including Google’s Privacy Sandbox. As a longtime collaborator with Google, we’ve met with them approximately twice a week for planning and strategy management over the last three years and participated in dozens of workshops where we delved deep on a technical subject and reviewed or created API design.
To prepare for the 1% cookie deprecation, our teams have worked tirelessly to ensure successful setups, including integrating with Privacy Sandbox APIs, such as Protected Audience, Topics and Attribution Reporting API. We also developed new and dedicated bidding models and bidding infrastructure, as well as a strategy for user shopping behavior collection and storage—adapting to Interest Groups in Protected Audience. Furthermore, we created new product recommendation mechanisms, adapting to Protected Audience, on top of new and dedicated reporting pipelines and infrastructure.
We’re proud of our innovation around this project, which has strengthened our leadership around Privacy Sandbox initiatives, and prepared us to run testing at the highest possible confidence levels. Here, it’s important to say that the quality of testing results will hinge not just on our innovation and methodology, but also on the size of the sample and how the numbers are standardized.
Achieving statistical significance
One of the earliest learnings we discovered in our test preparations is that scale is extremely important, as not all Privacy Sandbox testers are created equal. For instance, our extensive reach, spanning 18,000 advertiser clients, including 220 retailers, and over 1200 premium brand-safe, open internet publishers increases statistical significance with approximately 100 million weekly impressions. However, smaller DSPs or individual advertisers participating in the Privacy Sandbox do not have this reach, and therefore have far fewer testing outcomes that can contribute toward achieving meaningful measurement. And very few publishers will have independent traffic volumes to meet statistical significance thresholds.
When we look at these tests, 1% of Chrome traffic is still a small percentage (albeit representing approximately 32 million Chrome users), so smaller DSPs who are not purchasing ad inventory in large volumes have far fewer chances to gather data and may struggle with statistical significance. If DSPs have these struggles, then individual advertisers will surely face the same issue to an even greater degree. Considering the need for a scalable test setup, we’re actively testing and collecting enough data. Once we hit the required threshold, we’ll share all the details about our preliminary findings.
Another consideration to enable accurate and meaningful tests is to ensure that all participating DSPs have comparable setups. Setup discrepancies across participants can invalidate results and make them susceptible to bias. For instance, if DSPs don’t agree on how to treat users for which both third-party cookies and the Privacy Sandbox APIs are available, it can lead to inaccurate or skewed results.
So these nuances are not overlooked, we’ve been in close communication with Google and the CMA to ensure participating testers are aligned in their setup to avoid discrepancies, and we will continue to work with them throughout the testing period.
The Privacy Sandbox complements our addressability strategy
As I hinted before in this post, working with Google on their Privacy Sandbox initiative is just one pillar of our multi-pronged approach to solve for addressability, as we’re also focused on first-party data solutions and leveraging closed environments like retailers’ ecommerce sites and social channels. Marrying these diverse solutions with evergreen underpinnings of unifying product, contextual and other cookieless commerce signals through AI, allows us to empower our clients to achieve performance and customer acquisition outcomes through personalized advertising.
We believe that being prepared for potential changes in Chrome is essential for enabling the marketing efforts of our clients, and monetization for publishers. If the Privacy Sandbox moves forward as planned, we want to ensure it does so in a way that benefits our entire ecosystem. That is why we take these tests incredibly seriously.
Taking on testing together
The Privacy Sandbox tools are complex and difficult to implement. Given that we are already ahead of the methodology and have impactful scale, we welcome advertisers, publishers and other large-scale DSPs to partner with us to ensure their investments best suit their marketing objectives. The weight behind our results will speak loudly when shaping Google’s proposals, even though only a small percentage of Chrome’s traffic is available for testing, and we’ll continue to advocate for our clients across the industry.
Today, Criteo’s existing integration with Google Privacy Sandbox, as well as our ability to activate first-party data and deliver campaigns in closed environments, enables a well-balanced addressability strategy on behalf of our clients. For those partners with a standard Criteo integration, participation requires no further action on their side beyond ensuring proper implementation. Criteo campaigns will be automatically future proofed for addressability in Chrome. There is no need to decide which API gets used, as our optimization engine and best-in-class AI choose the right addressability solution at the right time to enable user-level or interest-based targeting based on available, privacy-protected information.
All of us will know much more about the future of the Privacy Sandbox in about six months, but in the meantime, advertisers should be evaluating if their tech partners meet the threshold for statistically significant results. This will let brands and retailers focus on creating and selling products and growing their companies rather than solving an incredibly complex technology challenge. Solving that issue is our focus every day, you can count on it.
Visit our Addressability Hub for more tools to navigate the next chapter of advertising without third-party cookies and bookmark it to check for ongoing updates throughout 2024.
