AB Testing Results

Which Experiment Won? The Results of A/B Testing

Howdy ,

August saw a change in tack for us as a business, we decided that we had reach a key place for UnderstandingE and felt it was about time we ran some A/B testing to see what worked.

A/B testing was a fun topic to cover in the weekly HappyHour meetup and we launched a Google experiment to see which colour scheme worked best for new registrations on the site.

This was really a case of throwing some ideas at the wall and seeing what stuck.

In this article I’ll be covering what was learnt and remember, this has a direct impact on you as we’ll be covering this in “Plain English” with you in the new year.

What to Test?

To start any experiment we needed to decide what to test.

We decided to stay with something that was simple, but would have a positive impact to UnderstandingE.

A week or so before starting the experiments we had implemented a pop-up join box for new people to the site to join. This seemed like an obvious thing to target as it would have a positive impact if we could improve the conversions of passing people into registered users.

Note: This would be slightly different for you as an eCommerce site owner. The buy button colour would have been my first target if we were running a purely eCommerce Magento website.

What to Change?

I wanted to keep the tests as simple as we could and the easiest thing for us to tinker with was the colour scheme that is being used in the popup window.

The standard popup is just black text on a white background, not very appealing at all. So while keeping the text pretty much the same for each page, the changes on the test was going to be the colour schemes for the pop-up windows.

Note: There were  some minor changes in the title that is displayed to focus in on the page type that is being viewed.

Not being very well colour coordinated, I found a handy resource that had 14 different colour schemes I could use here:

http://blog.crazyegg.com/2012/07/11/website-color-palettes/

I also went with some well known colour schemes, namely:

  • The NHS blue
  • John Lewis green
  • The Amazon grey/yellow
  • And the eBay blue

The NHS blue is designed for security & trust, John Lewis just because it was green that seemed to fit well with the site and eBay & Amazon because they were no-brainers to add as our target audience of marketplace business owners would recognise the colours.

So we had our colour schemes set, then what?

Google Experiments

google experimentsI had previously used the old version of Google Experiments on the LastDropofInk site and had some really good results.

The bad news was that since I last used them, they had been updated and were moved into Google Analytics.

After fighting with Google experiments to make a multi-variation test (which I can assure you we won’t be using again as it was a nightmare to set up), the experiment was set and then crucially left alone for at least a month to see what would happen.

When I say fight, I really do mean it. Setting up a multi variation test using Google Experiments was a utter nightmare and I finally worked out the code that was needed to make the colour changes switch using jQuery on the site.

I won’t bore you with the exact details, however the basics are really simple. On the popup window a function was called and it was this function that then called the Google Experiment and changed the colour codes accordingly in the popup. If you’re interested the source code is here if you fancy doing what I did ( although I’d not suggest it as it’s quite geeky ).

It was left alone for a month for numerous reasons, the biggest one as I didn’t want to tinker with it and also to give it a long enough time to get some decent stats in.

It’s now October and it’s actually been two months!

We have been super busy after all and I feel somewhat proud that I managed to leave this alone for so long. You know what it can be like with web stats, refreshing the page every 2 seconds to see how many new visitors have been on the site.

And the Results?

I couldn’t resist and took a look at the stats at the end of September and updated Facebook with a first glance. There was already a clear winner and below is the table from the completed experiment after almost 2 months of running:

experiment variations

The Winning Experiment I’ve blurred out the actual numbers for the sessions and conversions for privacy, however the final two winners were a yellow & red colour scheme and the John Lewis green.

Now in hindsight, a no-real-surprise the John Lewis green did the best for us, mainly because our website is green and that would fit in well, but the numbers do not lie.

So basically green won, doh!

A screenshot of the winning colour scheme is to the right, click the image for a larger version.

One of the things that I noticed was that the number of experiment sessions were grossly much lower than the actual site visitors, this was even after comparing the Google Analytics stats to our own internal stats to factor out logged in users.

And Finally What Was Learned?

Well for one we won’t be using Google Experiments again.

It was too complicated to setup and having to spend an hour (or two) researching, then testing a custom script to make the desired changes and then to see that the number of sessions really did not match the sites unique visitors counts.

Looking back, I’m quite dissapointed.

After all, how on earth could I explain to you how to do this on your site in a few minutes rather than a few hours?

Basically I chose Google Experiments over other tools such as optimizely because it was free and this case, free was not a good choice.

It’s worth noting that we would fall in the really expensive package for optimizely as we’re way above the silver threshold for unique visitors, which was what deterred me from using it in the first place.

This was our first step into A/B testing for the UnderstandingE site. I threw 14 colour combinations, arguably too many at the wall and was it really any wonder that a colour scheme that matched the rest of our site worked best? Green.

While it’s been fun to come back and revisit the experiment and that learn that green converts well on a green website, next time around we’re paying for optimizely.

Matt

Image Source: Tea, two sugars

2 replies
  1. VideoGuy
    VideoGuy says:

    Nice, I really like this kind of stuff!

    I am certainly not surprised that the green won as I personally feel more comfortable when colours kind of fit into the overall colour scheme. BUT…

    My question now is: Did the colour maybe win because green is overall a more stimulating colour or could it really be because it fits well with the theme? Thinking of it makes me realise again how complex the human brain is to be influenced by such subtle things as colour.

    I also had a look at the article you linked here and it seems more & more sites are going for pastel colours with very solid cuts.
    Kind of the “less is more” approach I guess, but so often I feel too many pastels make a site look bland or dull.

    Then again, I hope my site isn’t too flashy and busy…
    As soon as I’m ready I will put up a link in the forum so you guys can crit it. (I’d rather tone down a bit than irritate my customers’ eyes!)

    Reply
    • Matthew Ogborne
      Matthew Ogborne says:

      Howdy JP,

      I know what you mean, I have been looking closely at a large number of sites recently to see how they handle colour schemes and the common approach is to be quite calm on colours.

      I really like our green and brown combination, so don’t worry we’re not hanging anything drastic in the future.

      Quite a learning experience all round and a few minutes ago I had the pleasure of pulling out the testing code and before signing up to any further tools, we’ll work out a battle plan on what to test next.

      Matt

      Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *