How many ad variants do you need?

Somebody told you to test three creatives, so you asked for three creatives. They came back and they are different from each other: different opening, different music, different edit, one has a voiceover and one has captions, two are vertical and one is square.

One of them did better than the other two. You now know that. You do not know why, which means you cannot ask for another one like it, and next month you are back where you started, asking for three more.

The comparison may identify a stronger package without telling you which element helped. Decide whether choosing a package or isolating a component is the question you need answered.

Why does the number not help you?

Because the number of variants follows the question, budget and data available. A concept comparison and a single-element comparison answer different questions.

Two ads differing in one planned feature may support a more specific conclusion when the test design is sound. Ads with several differences can still be compared as packages. The result must be described at the level the experiment tested.

So the answer to how many is: as many as you can afford to run while changing one thing at a time. That is a structure, and the number falls out of it.

One variable at a time

Same body, different hook: keep the remainder of the creative consistent and use suitable allocation, metrics and a stopping rule. The opening is the planned difference; the design decides how confidently an effect can be attributed to it.

Same hook, different first frame. Identical video, but the still that appears before it plays is changed. The thing you are learning here is not about the ad at all, it is about what stops a thumb.

The same picture with a different sentence said or written over it. This is the cheapest test in the entire list, and people skip it because it does not feel like enough work to count.

Same everything, different angle. The video is rebuilt around a different reason to care, and this is the only one on the list where a new asset is required.

Notice that three of the four are edits rather than shoots. Most useful testing is done in the timeline, on footage you already own, which is why filming several openings on the day is worth more than any decision you make afterwards.

How do you know your last test taught you anything?

Write the finding down as a sentence before you look at the results. If you cannot fill in the blank, the test could not have worked.

The sentence is: this test will tell me whether ______ beats ______, with everything else the same.

Try that on the last set of ads you ran. If the true version is this will tell me whether ad A beats ad B, then the answer will be yes or no and nothing will be learned that survives into the next round. The test has produced a winner rather than a reason, and winners expire.

When is testing not worth doing at all?

When the numbers coming back are too small to separate the versions, which is most of the time for most small accounts, and nobody wants to say so.

Here is the mechanism, without any figures in it. Every result you see is the real difference between the ads plus a large amount of luck: who happened to be shown it, what time of day, what they were doing. When the amount of data is small, the luck is bigger than the difference. Split the same budget across more versions and each version gets an even smaller share, so you have made the noise worse and paid for the privilege.

With a modest budget, too many variants can leave too little data for a useful comparison. Estimate what the test can resolve before choosing how many versions to run.

What do you do with a modest spend?

A sequence. One change at a time, each one running long enough to mean something, in an order you decided in advance.

Start with the question most likely to change the next decision. The opening may be a useful candidate, but the offer, message or destination can matter more in a particular account.

A small sequence of well-designed comparisons can build useful evidence. Record inconclusive results and the conditions of each test; a before-and-after change is a weaker observation, not a permanent fact about the audience.

One caution. If you change the opening and the audience targeting in the same week, you have two explanations and no way to separate them. Whatever you are testing, everything else in the account stays where it was, including the parts that feel unrelated.

What to change on the next round

Take the ads you are running now and write down, in one line each, what is different between them. If the list runs past two items, you are not testing, and the next brief should ask for the same video with one thing changed rather than three new ideas.

What follows is a question of temperament: whether you would rather have a winner this month or a reason you can use for the rest of the year.

Sources and evidence

What here draws on research into online experiments and what is our own method.

Research context. Online experiments need suitable allocation, measurement, sample size and stopping rules. One planned difference is a simple way to isolate a creative question, while a broader concept test can answer which package performs better. Small or poorly designed comparisons may remain inconclusive.

Our method. Changing one variable at a time, starting with the opening, and running a slow sequence rather than a wide test on a modest budget, is this studio's practice for applying that principle to a small ad account.

References.

• Microsoft Research. Online Experimentation at Microsoft. microsoft.com. Randomisation and proper design support causal claims. Platform experiments need power, guardrails and instrumentation.
• Larsen et al. Statistical Challenges in Online Controlled Experiments. tandfonline.com. Online tests have power, peeking, interference and metric pitfalls. Not specific to video platforms.