A Web Designer’s Introduction to A/B Testing

A/B testing is the process of experimenting with different options in order to measure the relative success of each one. Where the web is concerned, A/B testing can make it clear which design choices are most effective; clear to the designer and (more importantly) clear to the client.

There’s often a basic dichotomy between web designer and client. While the designer wants to make a beautiful, usable website, the client is focussed on achieving business goals. Clients often try to bridge this gap by influencing design decisions, and end up giving suggestions that seem jarring to the sensibilities of the designer.

However, what if the web designer closed the chasm between design and business by tweaking the website to increase the rate at which it achieves its business goals? This article explains how web designers can use A/B testing to increase the conversion rates of their clients’ websites and, in the process, make lives easier for themselves.

Overview

Let’s quickly run down what we’ll be covering in this article.

• Introduction to A/B testing
• Why should you A/B test?
  • Client’s perspective
  • Your perspective
• What can you A/B test?
• What should you A/B test?
• The correct approach to A/B testing
• Do’s and Don’ts of A/B testing

Introduction to A/B Testing

A/B (or “variant”, or “split”) testing describes the testing of two choices and measuring which one is better at achieving a predefined goal. In the context of the web, A/B testing means splitting your users between two variations of the same element or page (A and B) and seeing which is more effective at achieving the website’s conversion goals. For example, splitting users between two Call To Actions, one of which says “Get started now” and the other “Start your free trial”, then measuring sign up rates.

A/B testing has been used by the medical community in clinical experiments for a long time now. In fact, the first recorded clinical trial was an A/B test. In 1747, scotsman James Lind (a physician in the British Royal Navy) divided twelve scurvy afflicted sailors into six pairs. He then provided these groups different variations of citrus fruit, vinegar, cider, etc. and through the results found that Vitamin C rich citrus fruits help in preventing and curing scurvy.

James Lind feeding citrus fruit to a scurvy-stricken sailor aboard HMS Salisbury in 1747 (Artist: Robert A Thom).
Source Originally from: A history of medicine in pictures, published by Parke, Davis & Co. 1960.

Aside: I’m almost certain the first A/B test was conducted when two stoneage men got together and decided to see what kind of stone was best suited to starting a fire, but then, I’m no historian.

Engineers who don’t understand design try and clamp it down to rules and formulae in the hope that they can then use these to recreate good design. Web designers (usually) detest submitting their art to such rigidities. The most famous case of a designer coming up against a culture of A/B testing was at Google. In his post titled “Goodbye Google” Douglas Bowman of StopDesign (currently the Creative Director at Twitter) outlines how a fanatic reliance on testing even the minutest of design decisions was one of the factors which led to him leaving the company.

Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. –Douglas Bowman

This guide in no way advocates fanaticism of any kind. Instead, I’ll show you how you can use A/B testing to make it easier to bring in clients, enjoy better interactions with them (with greatly reduced heartburn), and make more money.

Why Should you A/B Test?

For a few simple reasons. They are:

• To help your clients achieve their business goals.
• To increase your earnings.
• To prevent heartburn.

Let’s cover each of these separately.

To Help Your Clients Achieve Their Business Goals

Unfortunately, a lot of web designers get lost in the design aspect of the project at the cost of neglecting the business goals. While people do care about a great looking website, they’ll give you a lot more love if you can make them a great looking website that converts better than their industry average.

Additionally, this also is an excellent reason for potential clients to give you their business.

To Increase Your Earnings

Offering optimization services can improve the way you run your business – it’s another feather in your cap. Genuinely taking care of your clients’ business goals will allow you to command a higher rate.

To Prevent Heartburn

Here’s a quick guide to preventing heartburn when discussing design with your clients.

When your client approaches you with:

It has to be fancy but hip. And since our company has the word velvet in it, I’d like to have the intro have some piece of velvet flying all over the place with velvety music in the background. – Clients from Hell

You reply: “Ok sure, let’s just A/B test it.”

Or if your client should say:

You say: “Ok sure, let’s just A/B test it.”

Or, your client says:

I’m having a rather Oatmealesque experience with a particular client’s website. The latest ‘feature’ they have requested is that background music play automatically when the site loads. What should I say to gently convince them that this is a bad idea? – stackexchange.com

You say: “Ok sure, let’s just A/B test it.”

Word of Warning…

Be prepared to suck it up. A/B testing has this interesting tendency to throw up results that are markedly opposite to intuition and the design sense of a professional designer. Take for example this case study on A/B testing small ad images.

Ben of PlentyofFish A/B tested these two ads to see which one recevied higher CTRs (Click Through Rates).

The result? In his own words, “0.049% CTR vs. 0.137% CTR in favor of the s*** ad in Microsoft Paint”. The point is, it’s quite likely that the CTA color chosen by your client’s parrot or her tarot card reader will out perform what you had originally designed. In that case, be ready to suck it up and continue making your client more money.

What Can You A/B Test?

While you can A/B test almost anything on a website, here are a few elements which, when changed, give the most noticeable effect on visitor behavior.

Source: formstack.com

1. Headline
2. Secondary headline
3. Copy
4. Testimonials
5. Call To Action text
6. Call To Action buttons
7. Links
8. Images
9. Placing elements above or below the fold

The Correct Approach to A/B Testing

The correct way to test is as a scientist would do. Scientists use something called “the scientific way” (bet you wouldn’t have guessed that) which consists of the following:

1. Research and observe: Research the behavior of the users a website is targeted at and understand where they tend to bounce off the path leading to the goal. For example, you observe that a lot of visitors either don’t fill a required form or start filling it and then abandon it somewhere in between.
2. Identify the problem: for example “The signup form has too many fields leading to a high abandonment rate.”
3. Create a hypothesis: for example “Reducing the number of fields in the signup form will reduce the form abandon rate.”
4. Start the experiment: Based on your hypothesis, you create a variation in which you reduce the number of form fields. You split the website traffic 50/50 between the original and the shorter variation, and wait for the experiment to run until it has achieved statistical confidence (95% confidence is the accepted standard and your chosen software should report this). Remember to always test against the original (the “control”) at the same time so you can compare results. This way you’ll know whether a variation is better or worse than the original.
5. Analysis: You see whether the variation performed better or worse than the control.
6. Conclusion: If the variation performs better, you implement it for all users. If it performs worse, you learn that form length is probably not responsible for the high abandonment rate and go back to identifying the problem.

Setting up Your First A/B Test

Most of today’s web A/B testing tools work in the following way:

• Make changes on webpage using a visual editor (this will be called the Variation).
• Tweak the test settings.
• Insert JavaScript code in website where you want the test to run.
• Start test.
• Wait for results, based on which you take further action.

For the purpose of this article, I’ll show you how to setup a quick test using Visual Website Optimizer (Disclaimer: yes, I work here).

After signing up for an account, start an A/B test from the dashboard and key in the website you want to run a test on. The page will open up in the Visual Editor.

Make the required changes:

Tweak the test goals and settings:

Now all you have to do is add the JavaScript snippet in your website and you’re good to go.

Do’s and Don’ts of A/B Testing

To round off, let’s list a few guidelines.

• First and foremostly, let go of your ego. This here is an experiment, not a test against your design choices.
• Test only one thing at a time. If you test too many changes at the same instance, you won’t be able to identify what’s working and what isn’t.
• Wait for statistical significance to be achieved. Most A/B testing tools report a winner when the math shows that 95% significance has been achieved.
• Always test control and variation in the same timeframe on similar traffic. Testing control one week and variation next is a common mistake that should be avoided as the profile of visitors may change in that time.
• Make big changes. If visitors don’t notice the change, they won’t react to it and you’ll end up with no decisive results from your experiment.
• Ensure that changes are consistent across pages. For example, when you make a change in one product page, it should be reflected in all product pages.
• If your client is okay with it, try to test continually. One A/B testing win usually convinces skeptical clients to do so. The resultant warm smiles and bear hugs usually convinces the designer that it isn’t all that bad either.
• Don’t leave the results lying. Make changes immediately when the results are positive and use the negative results to build your own internal knowledge base. The lessons learnt will help you with other clients.
• Never assume anything. Just wait for the results.

A/B testing is a hugely involved process and this introduction serves to cover the fundamentals. With any luck, I’ve made it clear that A/B testing can not only improve any given design, but that it can be invaluable in communicating design decisions with your clients.

Thanks for reading, feel free to ask any questions you may have in the comments section.