The ABCs of A/B Testing

Page 1 of 3

Article ImageTwo weeks after its 2013 launch date, Electronic Arts, Inc. sold 1.1 million copies of its "SimCity 5" game, with digital downloads representing half of all sales. President Barack Obama's 2012 campaign website increased donation conversions by 49% and, ultimately, raised an additional $500 million. And Amazon famously changed its checkout button 6 years ago, increasing site revenue by $300 million. What's the common denominator in all three success stories and in countless other case studies across the internet? Each conducted A/B tests, which produced phenomenal results. And they're far from the fringe minority: 75% of strategic phase marketers rely on A/B testing to learn about customer behavior, per MarketingSherpa's 2012 "Website Optimization Benchmark Report."


A/B testing is the term used for randomly experimenting with a control variable (A) and an experiment variable (B) for the purpose of statistically testing a hypothesis. As it applies to online site design, it's the process of testing and comparing two similar versions (A and B) of one or more web/mobile pages to determine which ones perform better and produce better conversion rates-which can include sales, hits, leads, and click-throughs-among randomly sampled, but similar, visitors. A/B tests can compare like elements, content, or designs, but not necessarily; sometimes, a larger discrepancy between the items being tested can help with ruling out a specific approach or a departure from the norm.

Today's A/B testing technologies afford a nearly limitless ability to experiment with various approaches for providing useful content and navigation to users. Tests are no longer limited to a single page. Websites can now test functionality that exists on many pages throughout a site, such as the presentation of assisted navigation that allows visitors to filter their results while searching for content or products.

Many sites now perform multivariate testing, which is a more sophisticated type or variation of A/B testing that lets users test individual elements within a piece of content or design in order to understand the relative impact of each element, or combinations of elements, upon conversion. Multivariate testing can be most useful as a discovery mechanism. You can test a few variations of each of the elements (such as imagery or messaging) in order to see what element has more impact on conversion. It's desired by many because it produces more particular results and more intricate data. "In the past, it could take several weeks, possibly even a month, to coordinate all the components of an A vs. B test. Today, it can be set up and perfected in a day or so," says Jason Parks, owner of The Media Captain.

Bob Dufour, president of Fusion, says A/B testing allows today's digital marketers to change things in an evolutionary way and, ultimately, be more responsive to their customers. "A/B testing has evolved from relatively simple to an increasing use of multivariate designs that will allow you to have multiple changes or variations per test," says Dufour. "Instead of doing 10 A/B tests, you can do one or two multivariate tests, learn more quickly, and predict outcomes better."

(Image courtesy of Shutterstock.)

Page 1 of 3

Related Articles

Perhaps the only thing that changes more rapidly than technology in today's amped-up digital environment is the terminology used to describe that technology and its impact on consumers--and marketers. One recent example is the advent of the term "omnichannel" marketing, which many struggle to differentiate from another relatively recent term--"multichannel" marketing. Still, those who are most enmeshed in the field say there is a key distinction between the two, and it's one that will have an impact on marketers as they continue to seek ways of having a meaningful impact on the consumers they hope to engage. And, importantly, it's less about technology than it may seem.
In this age of analytics, marketers are data detectives. A/B testing can help resolve many of their quandaries related to online marketing. But, first, what is A/B testing? It is a process in which you choose the best-performing version of a webpage, by randomly displaying different versions of your site to visitors and assessing the performance of each variant against a desired metric (such as clicks or sign-ups). You can test by tweaking one page element (such as headline, call to action, or image) at a time, or you can test changes to several page elements all at once (the latter is referred to as "multivariate testing"). Thus, when you use A/B testing, you are not flying blind. You're letting data drive your design choices and decisions. Think of A/B testing as "the scientific method meeting online marketing."
Sure, posting trip photos, tweeting links, and networking with old co-workers is fun. But online citizens want to tap into the pulse of the world around them, belong to a community, and be recognized. "Social media has evolved into a personal online identity. Before, it was used to connect with friends and update your status," says Jesse Till, social media coordinator for Chatter Buzz Media. "Now, it defines a persona. It's a chance for a person to express and broadcast themselves to the world."