How Effective is Split Testing?

A Critical Look at How Split Testing Can Help–Or Hinder–Your Conversion Rate

Web design: what used to be creative guesswork now leverages one of the greatest benefits of modern marketing–the ability to test, integrate, and optimize with split testing software.

Many companies are boasting major conversion and website optimization improvements that they have achieved through split testing software. We’re certain that by this time–the beginning of 2017–you’ve heard about the effect changing the color of your CTA button can have on your conversion, or how the simple rearrangement of a header menu can turn an inconsistent trickle of traffic into a river.

Split testing utilizes Big Data, turning it into a window that digital marketers can use to see the perceptions of their website visitors. The popularity of split testing data makes sense–after spending a lot of energy and money on your website’s design and functionality, you'd want to continually improve or increase the ROI of that investment. And that’s where many marketers turn to split testing software.

What is Split Testing Software Capable Of?

Split testing software tests the effectiveness of individual elements in your web design and layout. Also called A/B testing, split testing creates alternate layouts of your web pages, making changes to your website’s content and then testing them against the viewership of different audiences. You can customize split tests to run at specific times, on specific pages, or be triggered by specific audiences.

“, originally called Booking, changed the name of their company due to split testing results, leading to an improved conversion rate of 6.6%”

Split testing has become the tool of choice for many digital marketers, notably used by industry giants Google and Amazon (Amazon famously used split testing software to significantly increase their annual revenue by simply adding a guest checkout button)., originally called Booking, changed the name of their company due to split testing results, leading to an improved conversion rate of 6.6%–double the percentage of their competitor’s.

Split testing software is built for the use of marketers, and is easy to use. For example, Adobe Marketing Cloud, Optimizely, and Visual Website Optimizer, 3 of the most popular split testing services, are equipped with a user interface (Google just launched it’s own testing tool, too). This means available editing tools–WYSIWYG, image fields, drag-and-drop–that can make design changes directly to your web page. Once you’ve connected your web page to the software, it automatically identifies design elements on the page, allowing you to easily make changes and track specific clicks.

Split testing
Image via Optimizely

What Can I Do With Split Testing Software?

Split testing software, as a tool, can be used to gain some incredible insights and inform profitable actions. But, essentially, it’s just a tool, and how effective it is completely depends on how effectively you use it.

A/B testing is just another MarTech tool that marketers have now, but just like every other marketing tactic, it requires a strategy and plan. Choosing the tool is fairly easy, but to get the lift that these big brands claim to get isn't a matter of just integrating Optimizely and running a couple tests.

Split testing software won’t tell you what to test–it won’t point out specific design elements or buttons, and it won’t tell you how to change them. It won’t tell you how long you should run your test, or what sample size you need to see long term changes in your conversion.

Here are some things to consider when using split testing software:

1. Test Length:

Cutting a split test short is a common mistake among digital marketers. Many run the test until they see significant results–maybe a few days after initiating the test–and then they immediately make the variation live on their website. This leads to false confidence and illusory results; after implementing the variation you may be surprised to see your conversion rate unchanged, or worse, drop. Your user’s behavior will differ over hours, days, weeks, even seasons–it’s doubtful you’ll collect accurate results within a few days.

2. Sample Size:

Amazon, Google, see massive traffic volume every day. Part of their enormous success from split testing comes from being able to take advantage of a large sample size of users. For the rest of us, we may not see such dramatic percentages in the responses of our users. Many marketers take sample size into consideration before implementing changes, invoking a minimum number of user responses before analyzing their data.

Split Testing Hazards: Performance & Mangling Your Website

A word of caution: Split testing is what we call a “client-side” optimization tool. What this means, is that the page is modified by the split testing software after it’s been rendered by the server. The HTML of your web page is modified at the DOM (Document Object Model–what programmers use to build pages on your website) level as it’s displayed.

What this means is that you’ll have the ability to change page layouts on the fly–something typically done by front-end developers–but this can lead to some unintended consequences if not carefully implemented:

  • Slowing down your website’s loading speed. It’s a good idea to only run a few tests at a time. Running too many tests simultaneously can result in a slower-loading website.
  • Some technologies, like Varnish and server-side caching, are no longer useful on web pages running split tests. This is because of the “client-side” functionality of split testing software, which can be a problem for websites with a high rate of traffic.

Split testing tools give you access to the DOM to change your design, content, and page layout, and if used incorrectly you can really mess up your website in a visible way. Especially if you’re uncertain with managing HTML, make sure you have a staging environment to test out changes before publishing.

Adding Split Testing to Your MarTech Stack: How Vital is It?

In modern marketing, analytics are vital. Part data analysis, part actionable tool, split testing is a hybrid software that combines observation with actionable direction. It’s data is wholly unique–providing insights that few other tools can report so clearly, operating silently in the background of your website.