Everything You Ever Wanted to Know about Split Testing
Okay, so maybe this isn’t going to be quite everything about split testing. I mean after all, in 1,000 words, there’s only so much I can talk about. However, I think that our readers will get quite a bit out of this examination of split testing. So, let’s get started.
What Is Split Testing Anyway?
If you already know what split testing is, feel free to skip this section. You probably won’t learn anything new. In essence, split testing means testing different versions of your advertising to see which one works best.
You do it by creating multiple versions of your advertising, sometimes with a single change, sometimes with lots of changes and then randomly assigning different visitors to the different pages. You then watch which one works best and voila, you know what to use for everyone.
Of course, split testing does have limitations. For one thing, if you change more than one thing on the advertising, you will never be certain as to whether or not the majority of customers picked one or the other for specific reasons.
In addition, there is a certain amount of variation which will naturally occur regardless of which version of your advertising customers see. This means that a small difference is probably not significant.
All that being said, split testing is one of the best and easiest methods to test advertising and it can be used for almost anything, from landing pages to e-mails and even adWords advertising. Here are a few tips to keep in mind when working with this method of consumer testing:
Test Your Split Test Software
The first thing you should do before you start running a split test is to test your split test software. In essence, this means that you put the same version on all pages and watch what happens. If you see a wide variation with one ‘version’ of your page, you know that there is a problem with how the split test software is displaying pages.
Be Wary of Testing More than One Thing at a Time
As I noted above, split testing works best when you test just one thing at a time. This means that you test different versions of your headlines, different page layouts and other things one at a time. Yes, it’s time consuming but if you try to test too many things at once, you’ll never be able to zero in on which change is making the biggest difference.
You Can Test Price Too
Another thing that many people don’t consider is that they can test various price points through split testing. This allows you to see which one resonates best with your target audience. Oh and remember that a lower price isn’t always going to get you more customers. Sometimes, the higher price will make customers feel more confident about what you have to offer.
You Can Test Your AdWords Positioning
Here’s another neat trick that many people don’t think of when it comes to split testing: try different AdWords positions. You may realize that you can split test different copy on AdWords for example. However, it is also possible to test positions 1, 2 and 3.
Google often displays multiple ads next to search results and the first position always commands a premium in the bidding. Even if you get a percent or two less in the number of clicks, if it ends up costing you half of what it would cost for the first position then it’s probably a good deal.
Don’t Be Afraid to Test Larger Blocks of Copy
While I don’t recommend completely redoing your landing page and then running a split test between the old and new versions, there is also nothing wrong with testing more than a single word or sentence. Yes, you can test whether the button labeled “order now” as opposed to “buy now” works best.
However, you can also test whole pieces of text and see which version works best. But when you do this, keep in mind:
You Can Perform Multiple Tests
The great thing about split testing is that you can perform multiple tests. This means for example that you could identify a particular piece of copy or splash page design that works best and then you can go on to test additional items. Perhaps one small change in the new copy will make things even better.
You Can Ignore Minor Changes In Conversion
A half percentage point or even one or two percentage point differences in split testing are pretty meaningless. There is a certain amount of randomness that needs to be considered when doing split testing. Sometimes, by simple luck of the draw, one version of your advertising will do slightly better than the other one.
On the other hand, when you do see more than a percent or two difference, you can be pretty safe in assuming that the difference is statistically significant.
Of course, it also depends on how many people saw the two versions of your ads. If you had a split test with 5 people seeing one version and 5 seeing another version, it’s not necessarily conclusive because your test group was so tiny.
Test the Most Important Thing In Your Headline
Think carefully about the headlines you test. One thing that almost never works is to make long winded headlines talking about lots of different things that your product can do for your customers. Instead, pick one feature and create multiple versions of your headline based on that.
Alternatively, you ht=”322″ />Alternatively, you could also test whether Apple might run a split test where they test headlines advertising the ease of use of the Macintosh as well as headlines advertising the fact that Macintosh is not as crash prone as Windows is.
Bottom line, you can pretty much split test anything and the testing should be ongoing, to continue to refine your stuff. Not to mention that there will inevitably be changes in taste as time goes on meaning that split testing is always going to be an ongoing project.
Well that was very interesting and I am looking forward to what you post next. I have heard about this split testing before but I have not actually done it yet myself. I understand it requires some kind of software to do this correct? This is a very interesting concept I wonder why someone didn’t think of this before now?
Yes, you do need software to do it. If you have a WordPress based site though, you can simply use a plugin. Best of luck.
I first learned about split testing when I signed up for a news letter from a well known internet traffic guru and web marketer. He proceeded to drip daily emails which essentially blew up my mail box. That wasn’t necessarily a problem because I learned many things including split testing. In fact, when I would read these very educational emails and articles, I often felt as though he was split testing his mailing list recipients via the newsletter because I would receive emails about the same thing presented in different ways.
It is very likely that he was indeed split testing his e-mail list and that someone else got a different version of the same e-mail. Good marketers are constantly testing and refining what they send out.
When I first learned that marketers conduct split tests, the first thing that came to mind was how tedious this process must be. I understand the importance of examining your analytics and studying the customers’ response, but if you are going for total accuracy, you can only really change one element at a time and split test those results. Think of how long it might take to identify a significant factor that makes a clear difference in response.
Nobody ever said that marketing was easy. However, if you keep at it, you do find out what works and what doesn’t and ultimately you end not needing to keep split testing the same things over and over again. You may also get to the point where you don’t feel the need to split test anymore because the results of your changes are becoming statistically insignificant.
Unless you choose some significant enough differences in your tests, your split tests may just leave you splitting hairs trying to determine which campaign was more effective. I think how you prepare a split test is just as important as the results of the test – the analytics. That said, I believe in getting feedback for different versions of ads or designs. I just would not spend an inordinate amount of time with split testing. You could become obsessed with the testing process and neglect other aspects of your business.
Well it depends what you focus on. Changing one word here or there probably won’t make much difference. However, I have seen dramatic results when you split test different subject lines.
I was so caught up thinking about how to split test copy writing or layouts that I didn’t realize one of the easiest (and maybe the most obvious and effective) forms of split testing is testing two or more different price points. When it comes to price, I’d say you can test as many different price points as you wish because no matter how many price points you test, the one variable remains, which is the price.
Well I wouldn’t go too crazy testing many different price points. Try 2-3 at first and then see what converts best. You can always keep tweaking as well but at a certain point, you need to accept that the job is done.
I have also heard split testing referred to as A/B testing. However you decide to call it, the way I like to describe it is that it is a way of working out which of two different options will be the most effective. When used for your email campaigns, that means A/B testing can help you decide which email will the best one to send to your subscriber list from two different options which are labeled A and B.
The only thing is that split testing doesn’t have to involve only A or B. It can have more than two options.
Using split-testing software is clearly a powerful way to increase your website’s conversion rate (that is, its ability to turn visitors into customers). I have read that many of the web’s most powerful companies, including Amazon and Google, use this technique. The split-testing software lets you carry out all the necessary tests, although such tests often take several weeks to finish. And that’s the problem I have cause I’m impatient. So this type of software would be good for me.
Good things do come to those who wait ;). But seriously, any market testing will take time. You can’t find out immediately how the market will react because you simply will not get that many people to drop by so quickly.
Everyone agrees that optimization and testing are important keys to success. And everyone thinks they know the best ways to use them. However, in my experience, testing is a double-edged sword. If it’s done right, the benefits pour in; but if it’s done wrong, you literally can drive a business into the ground. I often avoid analyzing my site traffic analytics because I am so focused on developing that I am concerned about wasting hours analyzing and not getting anything tangible done.
Many marketing folks still don’t calculate the number of visitors needed to run a test before starting the test. As pointed out by this and other blog posts you need to decide the required sample size before the test. This ensures that you don’t get bitten by the euphoria (or depression) bug when you see your first statistically significant result and save yourself some bad decision-making.
Good point. A sample of 5 customers is not statistically significant. Of course, the question then becomes, how small a sample does work? 50? 100? 500? I think that’s a function of how many people there are in the market overall.
In order to have any hope of getting the most out of your internet presence you are going to have to engage in split testing. By that, I mean all-out no-holds-barred split testing. That is the only way to achieve your maximum potential business wise. If you do it successfully you will find that your conversions, opt-ins and sales will increase greatly. If not, then at least you can learn from it and come back with a better testing plan.
But it is also possible to go overboard and to split test minutia. One needs to be careful not to get too carried away with it.
The question is: what do you test and improve first? You can’t test and improve everything at the same time. Marketing optimization is a continuous process. Of course, the best way to define what to test first is to know what the potential impact is of all elements you test – and improve. I think it’s safe to say there is a gap between what most people test and what they should be testing from an efficiency viewpoint.
I have come across apps that cater to those with little or no development expertise. The tools profiled in that report are attractive for their ease of use, straightforward implementation, and simple dashboards. Those tools generally work by dropping a line of Javascript into your site and changing the visitor experience at the browser level. While they’re easy to install and maintain, there are some potential disadvantages for power users looking to do beta testing, price testing, and other types of experiments.
Hi there would you mind letting me know which webhost you’re working with? I’ve loaded
your blog in 3 completely different web browsers and I must say this blog loads
a lot quicker then most. Can you recommend a good hosting provider at a
reasonable price? Thank you, I appreciate it!
If split testing is an on going process forever how do you ever figure out how to use it to your advantage? I am not even sure what it is doing when it is doing something. Slit testing is confusing and you have done your best to clarify what it is and how it works here in your post and you did an excellent job of it as well. Keep up the good work.
I love the fact that you can use this split testing for so much more than content you can use it for prices as well and that bottom dollar is what everyone wants to know. Thank you for posting this information I am looking forward to the best results in my on line career thus far. Keep the posts coming because these are very helpful.
My pleasure. So glad we could help. And remember, you can literally split test anything. You can even split test how you handle customers who have already signed up — half maybe get a newsletter and half just get advertising but less frequently. See who responds better.
Split testing has been to confusing for me in the past so I was leery at even reading your post but I am glad that I did because your post explained it better than any I have read before. Thank you for the work that you did on this post you can tell that you have put a lot of work into it and you know what you are talking about.
My pleasure. Anything specific you found useful?