The #1 Mistake People Make With Split Tests - Miles Beckler
split test

The #1 Mistake People Make With Split Tests

This blog post is about the importance of good data in your split tests.

Get this wrong and you will literally be shooting yourself in the foot with each split test.

Get this right and you will be able to make consistent improvements on your conversion rates all across your funnel, thus earning more money with the same amount of traffic!

On your path to success growing your own online business, you will learn (if you aren't sold on it already) that split testing is key.

Specifically, split testing your opt in pages, headlines, offers, etc. is the key way for you to constantly improve your results.

The simplest form of split testing is what we call A/B tests. You change one variable on a page and then you measure how many people make it through your call to action to the next page.

You only change one variable so you can have clean data that will identify whether that one change created an improvement in your conversion rate, or not.

If you change two variables in one test, you will never know which variable actually gave you the increase or decrease in conversions… But this isn’t about split test best practices, it’s about the biggest mistake most people make when split testing.

For this to make sense, you need to understand the concept of “statistical significance

Even writing that makes me have flashbacks of high school and college that I’m not totally enjoying… LOL

But this concept is paramount to your success as an Internet marketer…

Statistical Significance

All you really need to know about statistical significance is that the larger the sample size, the more reliable the data and results. Inversely, the smaller the sample size… The more unreliable the result.

In short, rookies often declare a winner before having statistically significant data… This is the #1 problem in split testing!

Basing decisions on bad data will destroy your funnel conversions!

Already, were in too deep of theory here… That's a lot of words about some complex and vitally important ideas.

To make sure this is crystal-clear for you… You will see a real world split test example from my site next!

You will see the split test set up and the data from 2 separate time periods that will show you through graphs exactly what statistical significance means… And it will show you why you need to base your decisions on statistically significant data!

Because you need to know what to look for in your split test dashboard to make the right decisions!


Great… Let’s get to it!

Real World Split Test Example

First, the test…

For me, on this site, my most important conversion point is the opt in for my email list. This is at

Within this page, I’m testing a new headline to see if I can increase conversions through the wording on the page.

The old headline, known as my control, stated: “FREE COURSE: Get My Simple 7 Step Blueprint That Has Made More Than $1,000,000 Online...”

split test control

The new headline I’m testing, my variant, states: “Free Course Reveals My 7 Step Blueprint That Took Me From Side Hustle to Million Dollar Business”

split test variable

Super simple split test, right?

There are really only a couple of things changed in the headline… Taking the “reveal” approach to the beginning and trying to meet the reader where there at (side hustle) and help them understand where the course will take them (million-dollar business).

So this was my hypothesis… That the new “path revealed”version would out perform the past version I was running.

I set up a new split test with Dave’s help from… I just emailed him the new headline and he got it all put into place for me!

Split Test Results

Let's get into the data so we can get back to the whole statistical significance idea!

By the 2nd day with 50 visitors through the test, the results looked amazing!

It looked like a dramatic victory was at hand from the new test variation.  and here is the trap that most new marketers fall into (that you are learning how to avoid, right now)!

Take a look at the split test data, below:

split test results 1

Out of the gates, the new variation was converting at 60% and the old variation was only converting at 20%… That is nearly a 200% increase in conversions. According to the split testing tool I use there was a 99.78% chance that this new variation would be the original.

I went to a couple of “statistical significance calculators” and entered in the data based on the number of visits and conversions… And they said “your test is statistically significant you can count on these results”

But I didn’t buy it…

With only 50 visitors split between 2 variations… 25 visitors each and less than 20 conversions total, I knew from experience that the future may bring a very different result than what I’m seeing initially.

It is impossible to have statistically significant data with such a small sample size!

So I waited patiently…

Hoping that my astounding results continued, but more importantly… Focused on allowing the data to accrue in large enough numbers that I could truly count on the results.

Today, 5 days after starting the test, I took a look back in on the data I’ve now had over 200 visitors through the test and over 70 conversions… About 3 times the data.

What is the result today?

Well… The results are much closer.

split test results 2

The new variation is still winning, but it is only out pulling the control by a measly 14%. Also, the “chance to beat original” has dropped from 99% confidence to only 75% confidence!

The most important key to split testing & "Statistical Significance" lies in this graph above.

Notice how the gap between them has narrowed significantly as the number of visitors and conversions continues to increase. This is completely normal in the world of split testing.

A rookie conversion optimization student will often times make the mistake of turning off the split test after seeing the 1st days results.

All of the tools said it was statistically significant… It was outperforming by almost 200%... How could that new variation not be the winner?

Well… This is why the biggest mistake that most people make in split testing is turning off their tests to early!

At this point, I’m still not done with my test… I felt it painted such a clear picture to help you deeply understand what statistical significance means that it was time to write this blog post.

I like to see a minimum of 7 days on a single split test. This allows me to balance out the differences in traffic between the weekends and the weekdays. Honestly, I prefer to week tests… 2 Saturdays, 2 Wednesdays, etc.

I also like to see a total of 500 conversions or a minimum of 1000 visits.

At this rate, it will take approximately 20 days of running this test for me to get the kind of data I am confident I can rely on.

One other quick note here… This little headline tweak may not have been the best example of a split test to start with. I’m literally testing a new tool and a new team, so I whipped up something quick to just get the ball rolling.

Jay Abraham has a quote… “Test screams not whispers”

What he means by that quote is to be sure you’re focusing your testing on the big things… A totally different offer, for example. A completely revolutionary headline, for example.

I’ve already run several big tests on what I’m giving away as my opt-in.

Last year, my best giveaway was my free Facebook ads case study  “FREE CASE STUDY:  How I Generated 13,943 Leads & 188 Customers For $888.84 With Facebook Pay Per Click  Advertising”

Facebook ads opt-in split test

But this new course has been out pulling that Facebook funnel for many months now.

So, all in all… I’ve been split testing screams and an occasional whisper.

But that’s really not the point here.

The goal has been to help you understand how deceptive the statistics can be when you 1st launch a new split test.

You now understand that patients and discipline with your split testing is key to avoid the biggest mistake of turning your split tests off too soon.

Waiting to see 500 conversions, 1000 clicks, or 14 days of data are all good targets for you to shoot for with your data, before making any decisions about which split test variation wins.

I hope this has been helpful in taking an abstract idea like “statistical significance” and breaking it down into some simple ideas with some easy to use guide points to direct your efforts moving forward.

Any questions for me, get at me in the comments!

If you know someone who would get value from this post… Please share the link with them directly or in a Facebook group, etc.!

4 thoughts on “The #1 Mistake People Make With Split Tests”

  1. Great post.

    What about testing paid promotion data like Facebook ads or Google ads?

    1. What metric should I look on?

    Currently I focus on the number of clicks each ads gets (testing one thing at a time).

    2. When creating a new offer for the very first time. What should I split test first? Ads image, ads copy or funnel copy or funnel design (simple Optin or quiz funnel)

    Thanks for this detailed post. I really appreciate it.

    Looking forward to hear from you.

    -Suraj Sharma

  2. Miles, if I recall correctly I've heard you say that you need about 1,000 points in order to make valid split test decisions. My question is, does that number vary depending on your niche?

Leave a Comment

Your email address will not be published. Required fields are marked *