You probably test more than you think. Ad platforms rotate creative automatically. Email tools split subject lines by default. Landing page builders ship with variant testing already switched on. Somewhere in your stack, something is running a test right now that nobody set up manually.
Then there is the other half of the work, where none of that applies. Most teams have never noticed the gap because nothing in the toolset points at it.
Split testing needs three conditions
Strip it back and an A/B test requires exactly three things.
Something has to be live, because you cannot split traffic to something that does not exist. A platform has to be capable of serving variants. And you need enough volume to reach statistical significance in a sensible timeframe.
Digital advertising satisfies all three by default, which is why testing feels like a solved problem.
Now run through the things that satisfy none of them. Packaging headed to a print run. A visual identity going onto signage, vehicles and stationery. Billboards and other out-of-home placements. Radio and podcast reads. A product name, before the domain and the trademark filing. Trade show booths and printed collateral. A campaign concept, before the production budget is committed.
Each of those gets decided once, produced, and then lives with the outcome.
Your testing coverage is probably inverted
This is the uncomfortable part.
The assets you can test tend to be the cheapest and most reversible. A weak paid social creative costs a few days of spend and gets swapped out. A landing page headline changes in ten minutes.
The assets you cannot test tend to be the expensive, permanent ones. A print run. A rebrand across every touchpoint. A name that becomes a domain, a trademark and a warehouse full of stock.
So in most companies the rigour is concentrated where the stakes are low and absent where they are high. Nobody designed it that way. It is just what happens when you let the available tools decide what gets measured.
Asking the team is not a sample
The default fallback is an internal review. Four options on a screen, a meeting, and whichever one draws the most nods.
The problem is not that colleagues have bad taste. It is that a handful of people cannot produce a reliable result even in principle. Eight people splitting five to three is statistically indistinguishable from a coin toss. You would never accept that evidence from an ad platform, and you should not accept it here either.
There is a second issue. Everyone in that room has watched the work develop for weeks, knows the brief by heart, and usually knows who made which version. None of that is true of the person who will see it for two seconds on a shelf.
What actually changed
Closing this gap used to mean commissioning a research agency: a scoped project, a six week timeline, and a budget that only made sense for large brands making large decisions. For a small team choosing between four label designs, that was never realistic.
A category of self serve platforms now runs the same basic exercise for a fraction of the cost and returns results within a day. Tools like goahead let you upload the competing options, define who should respond by attributes like age, location and gender, and get back a percentage split tested for statistical significance instead of a room’s impression.
How many responses you actually need
If you have run split tests before, this logic will be familiar. The sample size you need depends on how close the race is, not on how important the decision feels.
With two options, roughly 100 responses can confirm a split around 60/40. Around 500 can confirm 54/46. Around 1,000 can confirm 53/47.
Read that the right way round. It does not mean you need a thousand people. It means that if two options are almost tied, no realistic sample is going to separate them, and that is a useful thing to learn before you spend three more weeks refining the difference.
Three things worth being clear about
Preference is not purchase. Someone choosing a favourite in a study is not someone paying at a till. This is an input into a decision, not a sales forecast.
It only ranks what you give it. Four weak options produce a winner that is merely the least weak. No preference test tells you the whole direction is wrong.
And presentation has to be even. If one concept sits on a realistic mockup and the rest sit on a white background, you are testing mockups.
None of this replaces judgement. It just means the most expensive decisions in your marketing stop being the only ones made without any evidence at all.
Chris Mcdonald has been the lead news writer at complete connection. His passion for helping people in all aspects of online marketing flows through in the expert industry coverage he provides. Chris is also an author of tech blog Area19delegate. He likes spending his time with family, studying martial arts and plucking fat bass guitar strings.