What Is A/B Testing? How to Test So You Decide with Data, Not Gut Feeling

“I think the red button would be better.” “Let's try changing the ad headline, it'll probably work better.” Sentences like these come up in marketing meetings every day, but “probably” shouldn't be the basis for business decisions. A/B testing turns gut feeling into data you can actually prove.

A/B testing compares two versions (or more, in the case of multivariate testing) of the same thing, such as a web page, email or ad, by randomly splitting the audience so each group sees one version, then measuring which version delivers more of the desired outcome, such as a higher click-through or purchase rate.

The most important principle of A/B testing is to change only one variable at a time (isolate the variable), so you can be sure any difference in results really comes from that variable and not from other factors. If you change several things at once, such as the headline, image and CTA button together, you'll never know which one caused the change.

A/B testing that produces reliable results needs all the key components. It starts with a clear hypothesis about what effect you expect the change to have and why, followed by random assignment that splits the audience evenly, so no group has special characteristics that skew the results.

You also need a large enough sample size for the results to be statistically significant, and a long enough test period (usually at least 1–2 weeks, or a full purchase cycle) so the results aren't distorted by short-term fluctuations, such as weekends when customer behaviour differs from weekdays.

The most common A/B testing mistake is drawing conclusions too early, for example seeing that version B has a higher conversion rate than A after only 2 days of testing and rushing to conclude that B is better, when the difference may just be statistical chance.

Statistical significance measures whether a difference you see comes from a real change or is just random fluctuation. Marketers usually set a 95% confidence threshold before concluding that one version has truly won. If the test hasn't reached this threshold, keep collecting data before deciding.

Not every element has the same impact on results. High-impact elements to test first include the main headline, because it's the first thing viewers see; the call-to-action, both its wording and position; the main image or video used to grab attention; and the offer, such as a discount or free gift, which often affects decisions more than design elements do.

Things that usually aren't worth testing first include button colour alone or the font, because their impact is usually very small compared with the effort. Save them for later, once the main elements are settled.

1. Set a clear hypothesis about what you're changing, what effect you expect and why
2. Choose one primary metric to judge the result such as conversion rate or revenue per visitor, just one, rather than mixing several metrics
3. Calculate the sample size you need before starting the test, so you know how long to collect data
4. Run both versions at the same time in the same period, not testing A first and B afterwards, because external factors may change
5. Analyse the results once they reach statistical significance then roll out the winning version and record the results as a knowledge base for your next test

A/B testing turns marketing decisions from personal opinion into provable data. The keys are to test one variable at a time, collect enough data and wait for statistical significance before concluding. Testing regularly builds an ever-deeper understanding of your customers. If you want to start testing systematically but don't know where to begin, the Markettung team is happy to help design tests that hit the mark.