Why real incrementality starts where the platform ends
Attribution can be a mess. We all know it. Lately, everyone is treating incrementality testing like the holy grail that makes all our tracking problems disappear. Platforms like Google and Meta are handing out experiments that promise to show exactly how many “extra” conversions your ads are bringing in. It sounds scientific, looks controlled and feels causal. It’s great that these giants are making these tools more accessible. It’s a step in the right direction, as it gives every advertiser a chance to measure the actual value of their marketing. But there’s a catch.
The results of these tests depend entirely on three things: the setup, the data you feed it and the platform it’s stuck in. They give you a piece of the puzzle, but they definitely don’t give you the full, objective picture of your marketing’s impact.
And that’s exactly why testing outside the platform is the only way to see the real impact.
Catch #1: The algorithm is grading its own homework
When you run an in-platform test, you’ve got a test group and a control group. Sounds fair, right? Except the selection, optimization and attribution are all handled by the same algorithm that’s trying to sell you the ads.
You can almost hear the system thinking: “I want to show off as many conversions as possible. So, I’ll show these ads to the people who are already most likely to buy.” It then uses that same logic to "optimize" the campaign.
As an advertiser, you want a truly random test to see what happens with and without ads. In reality, that rarely happens. Because these platforms operate as a black box, you’re left with a system that likely tweaks variables in the background to seek out high-intent users. You aren’t measuring a clean, static experiment; you’re measuring a dynamic system that is constantly trying to prove its own worth.
Catch #2: The silo problem
Here’s the bigger issue: a platform can only see what happens in its own backyard. It has zero visibility on your TV spots, your retail promos, what your competitors are doing, or even just a rainy Tuesday that drove everyone to shop online. It only sees what happens inside the platform. That’s a problem, because marketing isn't a game you play in isolation.
Most performance channels are just there to catch existing demand. Search, shopping, and retargeting are great at scooping up people who already want to buy. But that intent didn’t come from nowhere. It was built by your branding, your video ads, or just the sheer strength of your brand.
When you run an in-platform test on branded search, the platform will show you an "uplift." But it can’t tell if that uplift came from a new customer you just created, or if you’re just paying for someone who was already walking through your front door. It measures platform effect, not market effect.
The solution
Out-of-platform testing changes the game. By using independent sources, like your own backend data, you move from platform metrics to actual business results. You stop looking at "platform conversions" and start looking at total revenue, new customers and actual margins. This is where it gets real, because it connects your marketing directly to your P&L.
When you test by completely turning off media in specific regions, and then look at your total sales data, you see the truth. You see the cross-channel interactions. You see the offline effects. You see the big picture because you’re the one holding the camera.
Out-of-platform testing forces you to look at marketing as a whole system rather than a collection of silos. It stops you from giving all the credit to the channel that happened to be the last one a customer clicked before buying.
An in-platform test asks: "What happens if I turn off this platform?" An out-of-platform test asks the much better question: "What happens to my entire business if I do this?"
If you’re serious about marketing effectiveness, remember this: platform data isn't the whole truth. It’s just a part of the story. The real truth lives in your integrated business data. That’s the only place you can see if you’re actually winning new ground, or just catching the demand that was already coming your way.