Do Parallel A/B Tests Contaminate Each Other's Results?

My big goal for 2026 was to hit what I call terminal testing velocity. Running as many experiments for a client as we physically can at any one time.

For years I assumed there was a hard ceiling on that. One test per page, maybe two. Any more and you dilute the traffic so badly that nothing reaches significance before the results go stale.

So the ceiling came down to maths.

But there was a question sitting underneath the whole thing that I'd never actually checked. If I put one visitor into two experiments on the same page at the same time, does the first test contaminate the second?

Most people assume yes. I assumed yes for a long time.

So I went and dug through the research to find out. The answer changed how we run our entire testing program, and it's the difference between running under 100 tests a year for a client and running over 200.

Why one test per page quietly kills your velocity

Let me start with the problem, because the maths is what traps most people.

Say a page gets 50,000 views a month. You run one test on it. Roughly 25,000 people go through the control and 25,000 through the variant. Depending on how many conversions you're seeing, that might be enough power to reach a result in a couple of weeks.

Now run two tests on that same page. The traffic splits three ways instead of two. Each variant gets around 16,500 visitors instead of 25,000.

Add a third test and it thins out again. At some point there just isn't enough traffic behind each experiment to reach significance in the window you need. And a test that drags on for months isn't reliable anyway, because people's behaviour shifts across the year. Seasonality, promotions, ad changes, competitor moves, all of it shifts the baseline underneath you.

So the safe play looks like this. Run one test per page, let it finish, move to the next.

But that caps you hard. Realistically you've got a homepage, a collection page, a PDP, a landing page and a cart. Maybe a price test and a shipping threshold test on top. That's around 10 tests you can have running at the very most, and once you factor in dev time it's more likely 6 or 7.

Multiply that across a full year and you land under 100 tests per client. I wanted us at 200.

Most agencies would look at 100 tests a year and be pretty happy. I'm not. That's not terminal velocity, and if you're trying to be one of the best in the world at this you have to push the ceiling higher.

So does running two tests on one page contaminate the results?

The whole velocity problem hinges on one assumption. That you can't put a visitor in two tests at once without wrecking the data.

I dug into this and the short answer is no, it doesn't, with one caveat I'll get to.

Microsoft ran a paper in 2023 called A/B Interactions: A Call to Relax. They looked at hundreds of concurrent experiments running on millions of users across four of their product groups, and checked every pair of overlapping tests for interference.

Three of the four products showed zero statistically significant interactions. The fourth had an interaction rate of 0.002%. That's about 1 in 50,000 test pairs. And there wasn't a single case where the overlap would have flipped a ship or no-ship decision.

So the idea that two tests on a page eat each other is basically an old wives tale. Nobody had really gone and proven or disproven it, so the assumption just stuck around.

And I get why it sticks. It sounds right. If you're changing two variables you should isolate them, that's testing 101. But at the scale real stores run at, the interference is so small you can't measure it reliably. Microsoft isn't the only one either. Booking.com runs over a thousand overlapping experiments at once. Google, LinkedIn and Microsoft all switched from isolated tests to overlapping ones years ago because keeping them separate was costing them more than it saved.

The one rule: don't test things that touch

The caveat is the part you actually have to get right.

The experiments can't be touching each other on the page. If two tests change the same element, or one is nested inside the other, they genuinely do interfere. That's not an old wives tale, that's real.

Microsoft's own example was an ad where one experiment changed the background colour and another changed something sitting inside that same ad. Those two fight over the same pixels, so of course the results get muddy.

In ecommerce this is easy to reason about. Split the page into zones and run one test per zone. Something changing the hero can't overlap with something changing the reviews, because they're nowhere near each other.

Test one thing in each zone and you're running four experiments on a single page with basically zero risk of them stepping on each other. If you want the full breakdown of how to structure this, I wrote a separate piece on how to A/B test your ecommerce store that goes deeper on the setup.

Why your testing tool decides whether this even works

There's a catch that surprised me. Whether you can actually pull this off comes down to your testing tool. The science says go for it, but the wrong tool will still stop you.

Parallel testing only works if your tool assigns visitors randomly and independently to each test. Some tools don't. They only let you mutually exclude audiences, which means a visitor goes into test A or test B but never both.

Mutual exclusion protects you from interference you don't need protecting from. And it splits your traffic to do it, which is the exact problem I was trying to escape.

Randomised participation works differently. Every visitor gets independently bucketed into every test they're eligible for. One person can sit in the control of one test and the variants of three others all at the same time. No traffic gets split. Every test drinks from the full firehose.

If you're on Shoplift you're a bit stuck here. It's a tool I still like, but its participation model becomes a natural ceiling on velocity. If you're on something with randomised participation you can actually stack. There's a Shopify specific wrinkle in how this plays out with templates that trips people up, which I covered in detail in running parallel A/B tests on Shopify with Intelligems. Worth a read if you're on the platform. If you're still weighing up which tool to run, I also put together a rundown of the best Shopify A/B testing tools.

One more thing on traffic. If you're a store doing a million views a month, you can afford to dilute and still hit significance, so mutual exclusion probably doesn't hurt you much. But most brands aren't there. For a store doing one to ten million in sales, the traffic to burn on splitting simply isn't there, and randomised participation is what makes high velocity possible at all.

What terminal testing velocity actually looks like

So run the maths again with parallel testing switched on.

You've got five pages worth testing. Homepage, collection, PDP, a landing page and cart. Run three or four tests on each one at the same time, respecting the touching rule. That's 15 to 20 experiments running concurrently, on the same traffic that used to support maybe five.

Over a year that clears 200 tests per client. Which was the number I'd been chasing the whole time, and it turned out the only thing standing between us and it was an assumption nobody had checked.

Why does the volume matter this much? Because for a new client our win rate sits somewhere around 15 to 25% early on, while we're still figuring out who their customers are.

We do a lot of research before we ever run a test. Psychographic profiles, scraping reviews across the web, retrospectives on past experiments, user surveys, the lot. That work matters. But the thing that teaches you the most about an audience by a mile is testing something and measuring what they actually do.

The more we test in those early months, the faster that win rate pushes up toward 30%. And with performance based pricing, that climb is the whole game. Slow testing means slow learning, which means we're leaving wins on the table for the client and for us.

For a store doing one to ten million, 200 well run tests in a year can genuinely change the trajectory of the business. You're lifting the store's conversion rate month after month, so paid ads get more profitable, which frees up margin to reinvest in growth. It compounds.

I spent years thinking one or two tests per page was a law of physics. Turns out it was just a habit built on an assumption, and the assumption was wrong.

Common questions about parallel A/B testing

Can you run multiple A/B tests on the same page at once?

Yes, as long as the tests target different elements that don't overlap. Two tests changing the same headline will interfere and give you unreliable data. Two tests changing separate zones of the page, like the hero and the reviews section, run independently with no meaningful interference.

Do concurrent A/B tests interfere with each other?

Almost never. Microsoft studied hundreds of overlapping experiments on millions of users and found an interaction rate of 0.002%, with no cases that would have changed a launch decision. The belief that concurrent tests contaminate each other is a myth in almost every real setup.

What kinds of tests actually interfere?

Tests that change the same element or nested elements. If one test changes a button and another test changes something inside that same button, they fight over the same part of the page and the results get muddy. Keep each test in its own zone and this problem disappears.

Does parallel testing need more traffic than running one test at a time?

No, and that's the point of it. With a tool that uses randomised participation, every visitor enters every test they qualify for, so adding a test doesn't shrink the traffic the others receive. Tools that only offer mutual exclusion split traffic across tests, which does slow everything down.

Can Shoplift run parallel tests on one page?

Not really. Shoplift uses a mutual exclusion model, so each visitor sees one test and adding more tests dilutes the traffic each one gets. For genuine parallel testing on a single page you want a tool with randomised participation, like Intelligems, where visitors can enter multiple tests at once.

How many tests can you realistically run on one page?

Usually three or four, one per distinct zone of the page. The real limit is how many independent, non-overlapping zones the page gives you to work with. Beyond that you start running out of elements to test without tests beginning to touch each other.

Sources

• A/B Interactions: A Call to Relax (Microsoft Experimentation Platform, 2023)
• Overlapping Experiment Infrastructure: More, Better, Faster Experimentation (Tang et al., Google, KDD 2010)
• Democratizing Online Controlled Experiments at Booking.com (Vermeer et al., 2018)