Most companies believe they test their advertising. In practice they launch two versions, check after four days which number is higher, and keep that one running. That is not a test, it is guessing with a chart. Here is how many variants make sense, how much money one variant needs, and how to read the result without fooling yourself.
Key takeaways
- Creative is the strongest lever you have. A 2017 Nielsen Catalina Solutions study built on roughly 500 campaigns attributes 47 percent of advertising driven sales lift to the creative itself.
- Test concepts, not details. Two genuinely different ideas tell you more than five shades of the same button.
- Three to five variants per test is the practical ceiling. Beyond that the budget splinters and nothing gets enough data.
- One variant needs at least 30 to 50 conversions before you can read anything from it. If you cannot reach that, do not test, decide on judgement.
- According to Think with Google, advertisers who used video experiments for lower funnel goals saw a 30 percent lower median cost per acquisition.
- A test running through a sale or a holiday measures the season, not the creative.
Why do most ad tests prove nothing?
Because too many things change at once. When two versions have a different image, different copy and a different audience, the result cannot tell you what worked.
This is the most common mistake and it does not come from ignorance. It comes from impatience: the company wants to know everything at once, so it pours everything into one test. Then version B wins, nobody knows why, and no rule can be drawn for the next campaign. That is the point of testing: not to win one test, but to earn a sentence that still holds in six months.
The second mistake is time. Four days is almost never enough. Ad systems have a learning phase during which performance moves on its own, and a weekend behaves differently from a Tuesday. Run a test for less than a week and you are measuring the day of the week.
The third is testing whatever is easy to change. A button colour changes in a minute, a new concept takes a week. So button colours get tested. But the gap between two concepts is usually several times larger than the gap between two shades.
How many variants and what budget does one test need?
Three to five variants, and enough money for each to collect at least 30 to 50 conversions. If you cannot fund that, skip the test.
The calculation takes a few minutes. Take your usual cost per conversion, multiply by forty, then by the number of variants. If one enquiry costs you 8 eur and you want three variants, you need roughly 8 times 40 times 3, about 960 eur for one test. With 300 eur in the budget, do not run three variants. Run two, or none.
This is where somebody has to tell the client something unwelcome. On small budgets it is more honest to make one good creative on judgement than to split the money across four variants where none gets enough data. A small budget test does not produce an answer, it produces the feeling that you decided on numbers.
Two technical points decide the rest. Every variant needs its own budget, or the system shifts money into whichever it likes early and silences the others. And let the test run untouched: every mid test edit resets the measurement.
What should you test first?
The first three seconds and the main claim. In that order, because what nobody sees cannot persuade.
The order we work through:
- The opening. First shot, first sentence. This decides whether the video gets watched at all, and the differences here are the largest on the list.
- The angle. The same claim framed as saved time versus framed as certainty. Not different words, a different reason.
- The format. A person talking versus a product demo versus text on footage. That is a concept test, not a detail.
- The proof. A number, a customer reference or footage from the floor. It checks whether people need a reason to believe you.
- The call to action. Only here. Changing button copy is the smallest lever of all.
How to arrive at those concepts is covered in our article on the creative campaign concept. Without a concept you are not testing ideas, only shuffling elements.
A practical note from production: when we shoot, we film two or three openings for the same video. It costs fifteen extra minutes on set and saves a whole second shooting day when the first version turns out flat. If you commission video externally, put it in the brief up front.
How do you read a result without fooling yourself?
Ask whether the difference would survive a repeat. Not which number in the table is bigger.
A two or three percent gap at a hundred conversions is not a result, it is noise. Repeat the test a month later and the other version may win. Deciding without statistics, use a blunt rule: take a gap above roughly twenty percent seriously when the conversion count is there, and treat anything below as undecided.
The second thing is the metric. Click through rate lies: a version with a provocative opening collects clicks and sells nothing. Measure whatever sits closest to money, an enquiry, an order, or at least a qualified visit. Which numbers are worth watching is covered in our piece on marketing measurement and KPIs.
The third is the write up. After every test, write one sentence: what we tested, what won, what we take from it. Without it you will run the same test again next year. After a year that list is worth more than any single campaign.
When does testing not make sense?
When you lack volume, when the problem is elsewhere, and when the test cannot change your decision.
The first case is arithmetic. A company with five enquiries a month learns nothing from testing, however good the tool and the consultant. There it is smarter to change creative more boldly and watch the trend across a quarter.
The second is a swapped problem. If people reach the website and leave, do not test the ad, test what put them off. Advertising can bring people in, it cannot explain the product for you.
The third case is the most honest one. If you know that whatever the result, the version the owner likes is going out, skip the test and keep the money. It sounds like a joke, and it is the most common reason tests change nothing in small companies. A test is worth running only if you accept in advance that your favourite version may lose. We lost a creative we were proud of exactly that way. The video ad creative experiments write up from Think with Google is a good sanity check on what a properly run test delivers.
Frequently asked questions
How long should a test run?
At least seven days so you cover a full week, and ideally until every variant has 30 to 50 conversions. A shorter test measures the day of the week and the learning phase, not the creative. If a sale or a holiday is approaching, postpone it.
Can I test creative without a paid budget?
Partly. Organic reach shows which opening holds attention, which is a useful signal and costs nothing. It tells you nothing about cost per conversion, because an organic audience already knows you. Treat it as a shortlist, not a test.
How many creatives do I need per month?
Fewer than the guides say, and more than you can realistically produce. For a small company two to four new concepts a month is sustainable, each with two or three openings. Regularity matters more than volume: creative wears out and performance drops even when you do nothing wrong.
Is an A/B test the same as an experiment tool?
Not quite. An experiment tool splits the audience so the versions do not contaminate each other, which is cleaner than running two ads side by side. If the platform offers it, use it.
If you are unsure whether you have the volume to test, write to us with two numbers: how many enquiries you get a month and what one costs. Within half an hour we will tell you whether a test is worth it, or whether the money belongs in one better creative instead. How we work is in our creative services, and you can reach us through contact.
