What is statistical significance?

Image Credit @justin_morgan

In any marketing scenario, optimization is the key. A consistent goal is to improve the process, create greater conversion or acquisition results, and spend less marketing dollars doing so.  

Optimization is one of those processes that link into everything we do, and everything we aim to do. As a business develops, and scales, so should the marketing efforts. The optimization process is fuelled by experimentation. 

Statistical significance is the technique trusted when shooting an arrow to ensure that marketers hit their targets.

If you shoot one arrow and you hit the bullseye, do you then assume that every time you shoot an arrow it’ll hit the same mark? Nope.

If you shoot 10, you’ll have an idea of your accuracy, but if you shoot 100, your understanding will be far greater.

Defining Statistical Significance

Statistical Significance is ensuring that a result that has occurred during an experiment or a campaign is not caused by chance. 

Creating a sense of reliability is pivotal in experimentation. When you have a high level of statistical significance, you can more confidently state that there is a relationship or correlation between the experiment conducted and the results found.

Statistical significance is about what you can hang your hat on, what you can justify, and how you can claim certain variables influenced the results. If you are hypothesis testing any level of your marketing, you need to back up the results with either a confirmation or denial of the hypothesis you tested. 

Understand your experiments

So how do you know that your experiments are valid? How do you know that your tests are making a difference? Could you just be pulling levers and hoping for the best? Are you just looking for confirmation bias? 

Any upper-level management will want to understand the results that are occurring from experimentation. Whether your experiment has floundered or created incredible results, they need to know why. They will also want to know if they can trust the data, was enough collected? 

They need to know you can take these results in three directions:

• Abandon the strategy
• Maintain the current course
• Scale up to ensure even greater success. 

Do I need to use statistical significance?

There are arguments for and against…

Yes - Especially so if you have enough data to work with to do so.

No - If you are unable to get enough data through for this experiment, at least partial data can help educate your decision-making instead of going on gut feel. If you do make a decision without enough data  however we would highly recommend circling back on this experiment and re-testing

Key Terms for Statistical Significance

Null Hypothesis:
The current state, or control element to the experiment. If you are testing, you’re investigating whether this current status can be improved upon, similar to a baseline.

→ Customers prefer the current headline of the landing page.

Alternative Hypothesis:
The difference made, the new addition, the alternative hypothesis is how you seek to search for results thanks to a new variable. This is the opposite to the control group, this is the trial group, who are given a new stimulus, a new piece of copy, a new image, or in our
example, the new headline. 

→ Customers prefer the new headline of the landing page

P-Value:
The p-value is the probability value of observing the effect from the samples chosen in the experiment. A p-value of <0.05 is the conventional threshold for declaring something is statistically significant. When considering your results, there needs to be a genuine difference in the results for it to be considered statistically significant, this is measured using the p-value. 

Sample Size:
This refers to the size of the sample audience who have been used for your experiment. The larger the sample size, the more confident that you can be with the results that come back. The more traffic that comes through the landing pages, the more likely the results will be a fair demonstration of user behavior or trends. 

Effect Size:
The size of the difference between the sample data sets, and this, more often than not, indicates practical significance. The effect size does relate to the sample size as a 0.5% increase to conversions when it's a sample size of 100 isn’t that impressive, but when it’s with 10,000 people, people begin to sit up in their chairs.

What to do when an experiment fails?

When an experiment fails, there needs to be an investigation as to why. This why is where statistical significance can come into play. This process is far better than the alternative of dumb luck that happens to have worked. There is sustainability when it comes to experimentation. 

A failed experiment has the next step. It’s understanding a journey toward success, rather than a lucky break, which can’t be explained. When you’re operating with a growth mindset, and constantly experimenting, the attitude toward results should be aligned with Thomas Edison’s approach to the lightbulb:

“I have not failed 10,000 times—I’ve successfully found 10,000 ways that will not work” - Thomas Edison

So by leveraging statistical significance, you’re improving the overall process of optimization within your campaigns and experiments, as well as communicating more clearly to your executives and upper management that your processes are driving a result.

With each result, you’re getting closer and closer to an optimized strategy that will offer increased conversions, less spending, or a better process, maybe even all three. 

When investigating your web experiences, or your signup form, you’re after the granular detail that tells you what is working. You want to be able to understand why the drop-off is occurring and subsequently run an informed experiment.

At Upflowy we live and breathe experimentation, we celebrate trying new strategies, and reinventing the playbook. If you’re looking to optimize the signup experience of your business, start experimenting with us today!