Picking A/B Tests That Actually Move Revenue

We ran an A/A test on a client's store last quarter.

For context, an A/A test is where you show the exact same page to two groups and see how long it takes for the testing tool to confirm there's no difference. It's a calibration check.

After running some calculations on their traffic and conversion volume, I figured out the test would need to run for over 100 days to be powered enough to meet the 95% confidence threshold. That's way longer than any store owner is going to sit around waiting for a result. But it also told us something pretty important about this store. If we wanted experiments to actually reach significance here, they had to be big enough and bold enough to create a measurable difference. Small UI tweaks weren't going to cut it.

We run a performance-based CRO model. If our experiments don't make the store money, we don't make money either. So burning three months on a test that shifts conversion rate by 0.3% is a problem for everyone.

We needed a framework. Something that would tell us, before spending any runtime at all, whether a test idea was worth pursuing.

Tiered Experiments

Here's the quick rundown on the framework. It's super simple. There are three tiers (Tier 1, Tier 2, and Tier 3).

Tier 1 - Unit Economics

Tier 1 changes are those that change unit economics. i.e. bundles, price changes, upsells, downsells and anything that will change the amount of money customers spend.

Tier 2 - Functionality Changes

Bigger functionality changes fall into Tier 2. These have tangible impacts on how customers use the website. i.e. mini cart changes, navigation changes, new discount opt-ins.

Tier 3 - Messaging & UI

Smaller changes that clean up visual friction or change the way the offer is pitched to a customer fall into Tier 3. They tend to take longer to reach significance and have a much smaller minimum detectable effect.

Why most experiments produce basically nothing

There's a study by Browne and Jones that looked at 6,700 A/B tests across a bunch of industries. 90% of experiments produced less than 1.2% RPV effect.

I think most teams are just testing the wrong category of change.

When we first started doing CRO, we spent a lot of time on what I'd now call Tier 2 and Tier 3 stuff. Cart drawer redesigns. Badge placements. Headline variations. Sticky add-to-cart bars. All felt productive at the time.

Some of it produced wins. But the wins were small.

We'd see a 3-5% relative lift, get excited, and then realise the confidence interval was so wide we couldn't actually be sure it was real. Or the test would need another 45 days to reach significance on a store doing 80,000 sessions a month. Pretty frustrating.

Then we ran a shipping threshold test for one client. Moved their free shipping cutoff from $75 to $99 and AOV jumped 22% in about two weeks.

I remember looking at the data thinking, why didn't we start with this?

It changed the economics of every transaction on the store. Not the layout, not the copy, not the badge placement. The actual money part of the equation. And the pattern kept showing up after that. The experiments that actually moved revenue were always the ones that changed what the customer pays, what they receive or how they decide.

That's what led us to build a classification system.

What makes something Tier 1

I use a pretty simple test for this. Does the experiment change the unit economics of a transaction?

Unit economics for an ecommerce store is basically AOV x Gross Margin x Purchase Frequency x Customer Lifespan, minus CAC. If your experiment directly touches one of those numbers, it's probably Tier 1.

So that's pricing tests. Bundle offers. Shipping threshold changes. Subscription vs one-time framing. Discount structures and BNPL messaging. Stuff like that.

These are the experiments store owners are usually too scared to run (which I kind of get, because changing your price feels risky). But I went and dug through the research on this and it's pretty one-sided.

Wharton studied 2,732 tests and found that price promotions produce the largest effect sizes of any experiment category. Intelligems ran over 1,000 pricing experiments and found 54% of stores discover a better price point. Median 6% gross profit lift.

For comparison, social proof badges produce about 2.3% RPV lift. Urgency timers sit at 1.5%.

The expected lift from a Tier 1 experiment is 15-40% relative. Tier 2 is around 8-20%. Tier 3 is 2-8%.

When you can only run maybe 8-10 experiments per year on a store (which is pretty common for brands under 200k sessions a month), the difference between picking Tier 1 tests and Tier 3 tests is often the difference between a client staying and a client leaving.

The scoring rubric we built

The decision tree is fine as a starting point but it's a bit blunt.

Some Tier 2 experiments outperform Tier 1 experiments. A quiz funnel that completely changes how people discover products can be massive. A tiny price adjustment on a low-traffic SKU might barely register.

So we built a scoring rubric. Six dimensions, each scored 1-5.

Transaction Proximity. How close is this to the moment money changes hands? Price test = 5. Homepage hero image = 1.

Degree of Change. How much is actually different? Completely new checkout flow = 5. Button label change = 1.

Visitor Reach. What percentage of traffic sees this? Sitewide = 5. A page that 3% of visitors hit = 1.

Behavioural Depth. Does this change how people think about buying, or just what they see on screen? Changing the value equation = 5. Cosmetic stuff = 1.

Evidence Strength. How much data supports this working? Pattern that's won repeatedly across your own tests = 5. Gut feel = 1.

Revenue Directness. Does this directly change revenue per transaction or does it affect something softer? Checkout conversion = high. Bounce rate = low.

Add them up. 22-30 is Tier 1. 14-21 is Tier 2. 6-13 is Tier 3.

We've been scoring every new experiment idea like this for about six months. It doesn't predict exact outcomes (nothing does) but it forces a conversation about whether something is actually worth running. And when you show a client that their "make the Add to Cart button bigger" idea scores an 8 while "test a subscribe-and-save option" scores a 26, the roadmap conversation pretty much sorts itself out.

Tier 2 and Tier 3 still matter though

I should be clear. I'm not saying ignore Tier 2 and Tier 3 entirely.

Tier 2 is your funnel architecture. How customers find products, how the cart works, checkout flow, subscription management, stuff like that. These are structural changes. Quiz funnels. Skip-cart flows. Mobile checkout paths.

We saw a quiz funnel on one brand drive somewhere between 80-469% more conversions than their standard navigation. That's a massive range but the point is it was always positive and usually by a lot (we've tested similar setups using different tools).

Tier 3 is everything visual and copy-related. Headlines, layouts, trust badges, image styles and CTA text. Easier to run, lower risk. Which is why most teams default to them. But the smallest effect sizes too.

The order is what matters. Start with Tier 1 on a new engagement. Get the big wins. Then Tier 2 for structural improvements. Then Tier 3 to refine.

On our performance contracts we aim for at least 60% Tier 1 in the first 90 days. After that we start layering in Tier 2 and Tier 3 work.

Calibrating this over time

The rubric is a starting point. It gets better as you use it.

After your first 20 experiments, plot your predicted tier against the actual result. You'll start seeing where it's accurate and where it's off.

Maybe your store's customers don't respond to shipping threshold changes. We've seen that with luxury brands where free shipping is already expected. Maybe your quiz funnel outperforms pricing tests because product discovery on the site is genuinely broken. Every store has its own patterns.

I went and looked at how other teams handle this and found some interesting stuff. GoodUI gets 71% prediction accuracy using pattern-based scoring (compared to 48% with no framework). Fresh Egg gets a 40% win rate with a similar dimensional approach. Industry average is somewhere around 12-20%.

The MECLABS conversion formula backs this up too. In their model, Motivation carries 4x the weight of any other factor. Value proposition is 3x. Friction and incentives are 2x each. So Tier 1 experiments are directly testing the things that carry the most weight. Tier 2 is friction. Everything else is Tier 3. The hierarchy lines up.

If you're putting together a conversion rate optimization program and trying to figure out where to start, I reckon scoring each idea like this and working from the top is a pretty solid approach.

Common questions about experiment scoring

How long should a Tier 1 experiment run?

Minimum 21 days to capture a full business cycle. Pricing experiments especially need time for purchasing patterns to settle. I've seen teams cut pricing tests short because early results looked good and then the numbers pulled back hard over the following weeks. If you're seeing a win at day 14, let it run.

Do I need heaps of traffic to use this?

It's actually more important with less traffic. If you can only run 6 experiments a year, picking the wrong ones is pretty devastating. For stores doing under 50,000 sessions a month I'd say you should almost exclusively run Tier 1 experiments.

What if the store owner is scared to test pricing?

This happens all the time. Store owners treat their price like it was carved in stone somewhere. But 54% of stores that run pricing tests find a better price point (Intelligems data from 1,000+ tests). I usually frame it as a learning exercise. "We're going to find out what your customers actually think the product is worth." The answer almost always surprises people.

Can a Tier 3 experiment ever produce a big result?

Sometimes. Swiss Gear ran a layout cleanup that produced a 52% conversion lift. But I wouldn't build a testing roadmap around outlier results. The framework is about playing the odds.

How does this work alongside ICE or PIE?

ICE (Impact, Confidence and Ease) and PIE (Potential, Importance and Ease) are good for ranking experiments within a tier. They don't predict magnitude well on their own because they're pretty subjective. I'd use both. Tier framework to classify, then ICE or PIE to prioritise within each tier.

Should I share the rubric with clients?

100%. It makes the roadmap transparent. When a client sees their button colour test scores an 8 and the subscribe-and-save option scores a 26, that conversation is already over. It also helps when someone's boss wants to test something specific (because that always happens). You can point at the rubric and go "we can, but it scored a 9 and we've got four ideas scoring above 20."

Sources

• Wharton, 2,732 experiments — Price promotions produce the largest effect sizes
• Browne and Jones, 6,700 tests — 90% of experiments produce less than 1.2% RPV effect
• Intelligems, 1,000+ pricing tests — Harvard Business Review, 2024
• GoodUI evidence patterns — 71% positive rate with pattern-based scoring
• Fresh Egg — 40% win rate vs 12-20% industry average
• MECLABS — Conversion formula: C = 4M + 3V + 2(I-F) - 2A
• Baymard Institute — 32 checkout fixes producing +35% aggregate CVR