Creative testing and audience testing

You changed the video and you changed who it was shown to, in the same week, because both felt overdue and running one test seemed more efficient than running two.

Something then happened, up or down, and you now have an explanation for it. So does everybody else in the conversation, and all of the explanations are equally supportable, which is the same as none of them being supportable.

If you compare only the old creative in the old audience with the new creative in the new audience, the effects are confounded. A deliberately designed experiment with separate combinations can address both, but that is a different design and may need more data.

Which question the test answers

Before anything is set up, write down the question in one sentence, and notice which of the two sentences it is.

"Does opening on the result beat opening on the problem?" is a creative question. The answer is useful for years and it changes what gets filmed. "Do people who already follow us respond differently from people who have never heard of us?" is an audience question. The answer changes where the money goes and it usually expires sooner, because audiences move.

Both questions can matter. For a modest test, isolating one is often easier to plan and interpret.

Why can a mixed test not be read afterwards?

Because the result has two possible causes and the data contains no way to separate them.

Suppose the new creative shown to the new audience does better than the old creative shown to the old audience. That is consistent with the creative being better. It is equally consistent with the creative being worse and the new audience being much better. It is also consistent with both being slightly worse and something outside the account, a season, a competitor, a news story, moving the whole thing.

Nothing in the numbers distinguishes between those. People resolve it by picking the explanation that matches what they already believed, which is not analysis. It is confirmation with a spreadsheet attached.

Which one do you settle first?

Settle the audience, then test creative against it.

A useful creative result can guide the next production, but its durability depends on the audience, offer and context. Keep the audience stable for a creative test when that answers the immediate decision.

The second reason is practical. Testing audiences properly needs several distinct groups running side by side for long enough to separate them, which spreads a modest budget very thin. Testing creative can be done inside one audience, so the whole spend is doing one job.

If you already have a defensible view of who your customers are, take that view, hold it still, and put everything into creative. That is the order that suits a small budget.

What does a clean creative test look like?

Same audience. Same placements. Same period. Same offer. Same everything, except one deliberate difference.

Choose a difference large enough to answer a useful question. Even a small wording change can matter, but the test needs enough data to distinguish its effect from noise.

Give each version enough of a run to be more than a rumour, and do not stop one early because it looks bad on day one. Early numbers swing wildly, and a version killed at the first wobble was never tested, it was just judged faster.

Then write the answer down somewhere permanent. The point of a test is not the campaign it was run in. It is that the next twelve videos get filmed differently.

When is a difference not a difference?

Most of the time, at modest spend, and this is the part nobody says out loud.

When the number of people involved is small, results bounce around on their own. Two identical versions of the same ad, shown to the same kind of people, will not produce identical numbers, and the gap between them is not information. It is noise wearing a decimal point.

Define the smallest difference worth acting on before the test, along with the metric and stopping rule. If the result cannot reliably distinguish the options, record it as inconclusive and choose on other relevant grounds.

What to write down before it starts

Four lines, before anything goes live. The question, in one sentence. The one thing that differs between versions. What is being held still. And what result would make you change what you film next.

That last line turns a test into a decision. If no outcome would change anything, the test is a way of feeling rigorous and the money would do more sitting in the better-performing version.

Take the last test you ran and try to write those four lines about it retrospectively. If you cannot, you now know what the next one has to look like, and that is worth more than whatever the last one appeared to show.

Sources and evidence

Research context. Work on online experimentation explains why allocation, measurement, sample size and stopping rules matter to causal conclusions. Isolating one planned difference is a useful simple design; it is not the only valid experimental design.

Our method. Settling the audience question before the creative question, and writing down in advance what result would change a decision, is this studio's practice for applying that discipline to a small ad account.

References.

• Microsoft Research. Online Experimentation at Microsoft. microsoft.com. Randomisation and proper design support causal claims. Platform experiments need power, guardrails and instrumentation.
• Larsen et al. Statistical Challenges in Online Controlled Experiments. tandfonline.com. Online tests have power, peeking, interference and metric pitfalls. Not specific to video platforms.
• Scheibehenne, B., Greifeneder, R. & Todd, P. M. Can there ever be too many options? A meta-analytic review of choice overload. academic.oup.com. Across 50 experiments the mean choice-overload effect was near zero and highly context-dependent. Does not establish that fewer options (or one link) reliably outperforms several.