Statistical Significance? Who Needs It? (Hint: YOU Do!)
December 1, 2017
We talk about testing. A lot. We talk about testing so much, you might think that all you need to do is simply create a test and you’ll be a PPC Hero yourself! Today we’re going to clarify how statistical significance is your key to proper testing and the true gateway towards being a PPC Hero.
Statistical significance, in manageable bite-sized terms, is the likelihood that the results you get from your test are going to keep occurring. Statistical significance can assure you that your results weren’t just chance, and there is an actual relationship between the winner of your test and the outcome you’re seeing.
Although you may be able to look at your data and say “Eh, I reckon she be a right winnin’ ad!” – there is a degree of assurance that comes with a quantitative process deeming that ad to be in fact a winner. Even though you see something increase (or decrease, depending on the metric), can you really call this one the winner? We know there are many factors that affect performance and using statistical significance of a winning test can at least narrow the validity of this assessment. You are able to reflect on the choices you’ve made on your client’s or boss’s behalf with the confidence only a mathematical formula can provide.
So, how do you find this magical number? There are countless calculators that allow you to simply input a few bits of data and in 0.2 seconds receive confirmation of your statistical significance. There are also more complicated calculators that ask for more data than the more rudimentary options. You are also able to create your own formula in Excel, but an easier source is provided by the Teasley calculator (a highly recommended tool for true statistical significance).
So let’s say you’re comfortable using a more basic option for determining the success of your test. Hot dog! Have we got the links for you! Like I mentioned, there are many free options for a simple calculation. If you are dubious of the true accuracy of the outcomes, Brad Libby certainly identified some of the common shortcomings of the widely used tools in this industry.
Before we launch into sample outcomes, let’s grab some data.
One example might be an ad test. When you pull the data from your ad test, you must first select your date range. Common best practice when conducting a test manually is to set yourself a timeline. Let’s take a look at an ad copy test I’ve been running for about 14 days.
I then go to one of the free statistical significance calculators. After putting in my click results and conversion volume, I see:
What?? No clear winner? My 188 clicks aren’t good enough??! In a word- NO. Although I can see that my Control ad brought in twice the leads, the volume of clicks we’re basing our assumptions off of just doesn’t give us full confidence that these results will continue once we’ve deemed the Test ad to be the big ol’ loser. It’s suggested that our sample size needs to be a bit larger to potentially yield a full-on winner.
So let’s try it again. I’ve got a display ad test that I’ve been running for only a week or so and it’s only brought in 17 conversions total, but I see that my CTR is quite a bit higher for my Control ad, but my conversion volume is clearly lower. Let’s see about this test.
For this test we’ll use another quick & easy tool for finding statistical significance.
Here we’re told our results are 91% certain that my control ads will continue to perform at a better rate than the Test ads. While 91% sounds good, most of us are going to push for a 95% confidence or higher. This test probably stands to run a bit longer and re-try.
Another sample might be through a more thorough tool provided by Cardinal Path.
For this test, you input not only your click and conversion volume but your impression data as well. You’re also able to select the confidence you’re looking for. Here we’re aspiring to a solid 95%.
This gives you an analysis of the CTR and the conversion rate as indicators of statistical significance. For those who are simply seeking to build traffic flow to their site, the CTR details may be the most helpful, where many businesses will look to conversion rate.Upon running the test, we can get some light analysis of what we’ve got going for us. My results from Cardinal Path’s calculator:
When looking at the conversion rate, Cardinal Path’s tool is telling us to hold our horses. Our data is not conclusive and there is not a statistically significant margin of performance between Ad A and Ad B.
So what do you do now?? You wrap your tests! Or at least, you check them for statistical significance so that you can present them to your client, your boss, or maybe just your mom with the confidence necessary to be a true PPC Hero!
Browse By Category
Becoming Allies With The Sales Team: A Lead Gen Solution
Collaborating with your client’s sales team can provide a treasure trove of insights. Explore how PPC tactics are utilized to help shorten the sales cycle and improve close rates!
The Opportunities Tab: AdWords vs Bing, A Friendly Match-Up
Dive into AdWords and Bing opportunities tabs and see how they compare. This includes accessibility, features, and performance insights.
How to Drive Calls that Actually Convert
Our experts are here to explain how gaining call tracking data can help you close the gap in your customer journey and tailor their experience so they actually convert.
A bi-weekly newsletter packed full of resources and strategies that will help make you a better PPC expert.
Hanapin Marketing | The PPC Agency of Experts Behind PPC Hero
New Facebook Analytics Updates from F8
Learn about journeys and automated insights, new tools announced at Facebook's F8 conference for the Facebook Analytics tool.
Spotify Advertising: The Why and How
This blog post gives you the rundown on Spotify, including why and how you might advertise on Spotify's new self-serve ad platform, Ad Studio.
5 Facebook Ad Settings to Audit In Your Account
5 Easy settings to audit in your Facebook Ads account to improve performance and understand website traffic.