Have you ever presented results from a marketing campaign and been asked, “But are these results statistically significant?” As data-driven marketers, we’re not only asked to but also to demonstrate the validity of the data — exactly what statistical significance is.

While there are several free tools out there to calculate statistical significance for you (), it’s helpful to understand what they’re calculating and what it all means. Below, we’ll geek out on the numbers using a specific example of statistical significance to help you understand why it’s crucial for marketing success.

In marketing, you want your results to be statistically significant because it means that you’re not wasting money on campaigns that won’t bring desired results. Marketers often run statistical significance tests before launching campaigns to test if specific variables are more successful at bringing results than others.

Statistical Significance Example

Say you’re going to be running an ad campaign on Facebook, but you want to ensure you use an ad that’s most likely to bring desired results. So, you run an A/B test for 48 hours with ad A as the control variable, and B as the variation. These are the results I get:

Ad	Impressions	Conversions
Ad A	6,000	430
Ad B	5869	560

Even though we can see based on the numbers that ad B received more conversions, you want to be confident that the difference in conversions is significant, and not due to random chance. If I plug these numbers into a chi-squared test calculator (), my p-value is 0.0, meaning that my results are significant, and there is a difference in performance between ad A and ad B that is not due to chance.

When I run my actual campaign, I would want to use ad B.

If you’re anything like me, you need more explanation as to what p-value and 0.0 mean, so we’ll go through an in-depth example below.

1. Determine what you'd like to test.

First, decide what you’d like to test. This could be comparing conversion rates on two landing pages with different images, click-through rates on emails with different subject lines, or conversion rates on different call-to-action buttons at the end of a blog post. The choices are endless.

My advice would be to keep it simple; pick a piece of content that you want to create two different variations of and decide your goal — a better conversion rate or more views are good places to start.

You can certainly test additional variations or even create a multivariate test, but, for this example, we’ll stick to two variations of a landing page with the goal being increasing conversion rates. If you’d like to learn more about A/B testing and multivariate tests, check out "."

2. Determine your hypothesis.

Before I start collecting data, I find it helpful to state my hypothesis at the beginning of the test and determine the degree of confidence I want to test. Since I’m testing out a landing page and want to see if one performs better, I hypothesize that there is a relationship between the landing page the visitors receive and their conversion rate.

3. Start collecting your data.

Now that you’ve determined what you’d like to test, it’s time to start collecting your data. Since you’re likely running this test to determine what piece of content is best to use in the future, you’ll want to pull a sample size. For a landing page, that might mean picking a set amount of time to run your test (e.g., make your page live for three days).

based on my alpha (in this case, .05) and the degrees of freedom.

Degrees of freedom are based on how many variables you have. With a 2x2 table like in this example, the degree of freedom is 1.

In this case, the Chi-Square value would need to be equal to or exceed 3.84 for the results to be statistically significant. Since .95 is less than 3.84, my results are not statistically different. This means that there is no relationship between what version of landing page a visitor receives and the conversion rate with statistical significance.

8. Report on statistical significance to your teams.

After running your experiment, the next step is to report your results to your teams to ensure everyone is on the same page about next steps. So, continuing with the previous example, I would need to let my teams know that the type of landing page we use in our upcoming campaign will not impact our conversion rate because our test results were not significant.

If results were significant, I would inform my teams that landing page version A performed better than the others, and we should opt to use that one in our upcoming campaign.

Why Statistical Significance Is Significant

You may be asking yourself why this is important if you can just use a free tool to run the calculation. Understanding how statistical significance is calculated can help you determine how to best test results from your own experiments.

Many tools use a 95% confidence rate, but for your experiments, it might make sense to use a lower confidence rate if you don’t need the test to be as stringent.

Understanding the underlying calculations also helps you explain why your results might be significant to people who aren't already familiar with statistics.

If you’d like to download the spreadsheet I used in this example so you can see the calculations on your own, .

Editor's Note: This blog post was originally published in April 2013, but was updated in September 2021 for freshness and comprehensiveness.