Split Test AcceleratorSplit Test Accelerator

Is Your Split Test Statistically Valid Yet?
Find Out Here And Now For Free.

If you've been split testing two pieces of advertising, and you want to see how significant your results are so far, you've come to the right place. There are four simple steps.

1. Enter your actions and impressions for ad A.
2. Enter your actions and impressions for ad B.
3. Press Submit.
4. Interpret your results (see below the form).

SPLIT TEST STATS

Ad	A	B
Actions
Impressions
CR %	0.0%	0.0%
Confidence

INTERPRETATION

Suppose the column highlighted in yellow has a confidence measurement of 99%. This means that there is a 99% chance that your higher converting ad will be better than the lower converting ad in the long run (like after a billion impressions or so). To be absolutely sure, you would have to run the billion impressions. But, since no one can afford that kind of certainty, we estimate much earlier and live with a little uncertainty.

You probably want the number to be greater than 95% before you put a lot of stock in it.

If your number is 95% that means that, if you go with the winner, 1 time in 20 you will be making a mistake. Most people can live with that.

With a confidence number of 99%, there's only a 1% chance that you'll be making a mistake by going with the leading ad.

Please note that your baseline confidence is 50%. In other words, your confidence can't go lower than 50%.

That puts a 70% reading in better context. I mean, suppose you get a reading that says you should be 70% confident. Without realizing the baseline is 50% you might think "wow, 70% confident, that's pretty good."

But really when you realize the possible range is from 50% to 100%, you realize that 70% is closer to the bottom than the top. 70% is, in fact, a very low confidence reading.

* NOTE: I must mention Brian Teasley's similar tool at http://www.splittester.com/ When adding a confidence number to the Split Test Accelerator I asked Brian about his tool, and he told me the general statistical method he used. I used that information to create the confidence measurement in the STA, and to create this tool. The free tools are similar, but there are some differences. The tool on this page requires different input than Brian's, and the tool on this page reports precise confidence numbers for all input. Beyond that, I believe the unerlying mathematics to be identical. Feel free to use the one that is easiest for you.

Here's another free tool that helps you determine how large your sample size should be for a given level of confidence and error interval.