Conversion Research and Testing 2/2 – Review

User Testing

How user testing differs from A / B testing

• A/B testing involves hundreds if not thousands of people, while user testing can be done with 10 people or less.
• Usability tests dont need a running online website – could be to each milestone point of the designing process
• a/b tests is with users on a live website even they dont know to be part of the test
• sample size: A/B testing involves hundreds if not thousands of people, while user testing can be done with 10 people or less
• User testing provides additional insights that A/B testing cannot, like emotional responses and fast detection of issues.
• A/B testing will give you statistical data that X is working better than Y, while user testing will tell you WHY something doesn’t work, HOW they would like to use the site and WHAT exactly is causing friction. It can be a much more insightful learning process.

When to conduct user tests

• Whenever you start optimizing a new website
• When you’ve done a design makeover for a website, but before you make it live (note that when creating a brand new design, keep usability principles in mind from the get-go)
• Whenever you change a critical part of your website (check the checkout, change category filters etc).

Test your actual target audience whenever possible

• Usertesting.com is a website for usability testing but be careful if they are not your target group because they might not figure out what the site is about. So be careful with specialized products
• for products about shoes or clothes it could make sense to have a broad spectrum of audience.

Sample size

• Minimum 5 and maximum 15 people
• Ten people will find 95% of your problems. 15 will find basically all of them.

• With user tests you can not have the conclusion for a statistic signification, therefore ist importatn to validate the hypotheses with split tests
• Only change parameters on the website when it´s a clear trend from the a/b test and the user tests

Running the tests:

1. Over the shoulder testing

• Ideal:
  • lab envireonment with different cameras, eye-tracking, facial coding capabilities, and heart rate and galvanic skin response monitoring to quantify perception, emotion, and arousal as they complete tasks on the site. You get important insights from:
  • see: during task-based usage, what areas are the viewing, for how long, and in what order;
  • feel: how do they perceive the site’s relevance, trust, orientation, security, & convenience.
• Disadvantage:
  • When user feels observed they may not feel free in doing the tests
  • Optimal lap environment could be expensice and time consuming

2. Unmoderated or moderated remote testing
   • Resume: I guess for me the best test environment including the cost-benefit analysis is to run a remote user test and give a short introducion before the user starting the test. Ist for better understanding and no misinterpretation to the questions. So the user will not lose time to guess what to do..
   • Record the session with tools like http://www.telestream.net/screenflow/, https://camstudio.org/ or https://www.techsmith.com/download/camtasia/
   • Share your screen with  https://www.join.me/de

• Dont ask yes/no questions – you want deeper insights / knowledge what are the friction points the user is confronted with

Tools (for mass market websites)

• https://www.trymyui.com/
• http://usertesting.com/

Tools (for niche audiences)

• your blog readers,
• newsletter subscribers,
• Twitter followers,
• friends of friends,
• using tools like Ethnio.com.

Tools (for remote testing)

• http://www.openhallway.com/
• https://www.trymyui.com/
• https://www.loop11.com/

Funnels and Goal Flows

• If there are funnels set up before you came along, don’t trust it – verify (conduct Google Analytics health check). It’s the optimizer’s goldmine.

Funnel for an ecommerce site:

Some thoughts:

• Last step – where actual payment happens – has “only” 79.44% conversion rate. I know from experience that 90% range is very common for ecommerce. That’d be definitely where I’d start. If I could increase 79.44% to 90% here the site would have 3711 orders instead of 3316. 395 extra orders would be ~$108,743 USD per month (I multiplied 395 with average order value of $275.30).
• My 2nd optimization priority would be Enter Email page. They’ve shown interest in proceeding from the cart by going to the next step, but abandon process. Something is causing too much friction there. Could there be a forced registration issue we could eliminate? Improve signup forms? If we were to widen the funnel in that step from 54.58% to say %65 – what would that be in dollars? You can calculate the impact!

Goal flows:

• You can see actual user flows toward goals, no backfilling (see link above)
• You can segment the flow by traffic sources, landing pages, events, technical data like browsers and so on

Google Analytics funnel visualization comes with 2 shortcomings (neither exists for Goal Flows):

• they don’t work retroactively,
• you can’t segment funnels in GA.

There are tools that can do both – like Heap Analytics, Mixpanel, Kissmetrics. Note that these tools are supplemental, and not a replacement to digital analytics tools.

There’s also a free tool that does it, called DashRocks. It connects to your Google account and pulls information from Google Analytics, and visualizes it for you. Also works retroactively AND you can segment funnels.

I use it for almost every project for its segmentation capabilities.

Here’s what it looks like for one of the sites I’m working in (“new visitors” segment applied):

Reverse Goal Path

Reverse goal (Conversions > Goals > Reverse Goal Path) path shows you the last 3 pages they visited before conversion. These are actual user paths people take before filling out a form or whatever.

Here’s one example:

A/B testing mastery

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”

Jeff Bezos, CEO Amazon

• Apply A/B testing in your companyto make sure you’re making the right decisionsand speed up and have trustworthy decisionsand not just expert opinions and stuffthat’s down this pyramid.That is the real value of A/B testing.It helps you to make better trustworthy decisions.
• A/B testing for research, lean deployment and real deployment
• looking for negative signals, for wins and for impacts or flood line to understand what needs to be done

• Statistical power is the likelihood that an experimentwill detect an effect when there is an effectto be detected.
  • So if you’ve created something good as a challenger something that in reality makes an effect
• Depends on:
  • Sample size
  • Effect size
  • Singnificance level
• So control is what we call the null hypothesis (the original state) -> Control = null hypothesis
• and the challenger is the hypothesis (the H0 against the H1) -> Challenger = hyptohesis
  • testing the null hypothesis against the hypothesis,
  • testing the control against the challenger

• Statistical power is the likelihood that an experimentwill detect an effect when there is an effect to be detected.
  • Really important to understand statistical poweronce you start doing A/B testing
• Depends on:
  • Sample size
  • Effect size
  • Singnificance level

Research to get insights for your A/B tests

1