Kinds of Split Testing Software

I read once about a man who tripled his conversion rate by changing a single letter in his headline. While that kind of improvement is not common (not by a long stretch), you can quite probably double your conversion rate after testing 10 or 20 items on your page. With that said, there are many things to consider when getting started with split testing. There are many differences among the split testing software options on the market today. Let me give you a brief taxonomy of the subject matter. The first main division is between simple split testing (single variable) and multivariate testing.

With single variable testing you just pick one element on your page to test. Take your headline, for example. If your current headline says "Are you having trouble with tooth decay?", you might want to see how "Do your kids have tooth decay?" works. In a single variable test, you just find a way to rotate each of those headlines on your page, and keep track of how many sales each generates. Paul Hancox has a simple split testing tool. And Google has one that's free, though it takes a bit to set up all your accounts to use Google's tool.

And the reporting is not real-time. But simple split testing can be discouraging. When you test a new idea against your control version, sometimes the control version wins. In fact, it often wins. You can get a string of disappointing tests all in a row. What would it be like if you could test 10 things at once? Well, you'd be bound to find at least one or two ideas that led to improvement.

And what if you could test all those ideas simultaneously with the same amount of traffic it takes to test one idea with simple split testing? Well, it sounds like magic, but that's exactly what multivariate testing can do for you. There are further distinctions among multivariate split testing tools. There are hosted solutions, and solutions you install on your server.

Typically a split testing program can run a little more quickly if it's on the same server as your webpage. Installed programs tend to be less expensive, too. You also will have complete control of your data.

On the other hand, with a hosted solution, you don't have to do an installation. Hosted solutions include Google's Website Optimizer, and Vertster. Installed solutions include the Split Test Accelerator and Kaizen Track. Taguchi testing is only one kind of multivariate testing.

There are also "full factorial" tools, and random independent rotators. When you run a Taguchi test, you run only a small fraction of the possible combinations of your factor levels. You might, for instance, run only 18 pages to test 4374 combinations.

This is made possible by orthongonal arrays. This allows you to find the best combination with out running through all of them. You do have to be careful about your factors interacting negatively with each other, but if you take care as you design your test, you can avoid the bulk of the problem with these.

The Split Test Accelerator, Vertster, and Kaizentrack are Taguchi testing tools. Full factorial tools allow you to cycle through all the combinations for your factors. So if you have 3 headlines, 3 offers, 3 P.

S. statements, and 3 images you're testing, you will cycle through 81 pages. One advantage of full factorial testing is that you don't have to worry as much about interaction effects.

Good and bad interactions should show up in the data. However, if you want to receive this advantage, you have to run many times the traffic through your test as you would with a Taguchi test. You also can't run as many combinations. Google's website optimizer is a full factorial tool. The Taguchi and full factorial tools both use arrays to maintain a relationship between the options being served for one factor, and the options being served for another.

Some split testers don't do this, though. They simply treat each factor as independent from the other factors. This leads to more flexible test designs, but also to longer and less accurate tests. There are other distinctions and features worth noting. And each solution has a different mix of pros and cons.

But this should get your oriented a bit before you make your decision.

Dominic DeLong has used taguchi testing tools for years to improve web sites.