Pitfalls of User-Testing: Learn Them Here
Saturday, December 03, 2011
As someone who always dreaded "market research" and data, I have to say that over the years I've become much more appreciative of marketing metrics. Most importantly, measuring consumer behavior, online and offline, has radically improved my marketing results. Enter the concept of user testing. In today's guest post Jason Thai explains user testing and the more common pitfalls associated with it.
User-Testing Overview
Anytime a web-based business
releases a new feature, several issues pop up, keeping product managers tossing
and turning at night:
·
1. Are
there any glaring bugs with the new feature?
·
2. How will
customers view the new feature?
·
3. Did we
design the feature in the most customer-friendly way?
·
4. Does the
new feature impact other unidentified areas in the site?
·
5. And most
importantly, how will the feature impact overall conversion?
If only there was a way to
test your products before launch… Luckily,
several companies have built fantastic user-testing platforms, providing companies with a pool of unbiased
respondents to provide feedback as if they were real customers. A lot of the sites offer great features such
as video walk-throughs with step-by-step user interaction. Another great aspect of user testing is that there
are few constraints on what you can test.
A few of the tests we’ve run include:
·
Testing
preferences for before and after web
page designs
·
Testing
the ease of our order flow process
·
Testing
the usability of new customer features
Also, user-testing is
relatively inexpensive, making it a no-brainer for identifying pitfalls prior
to product launch.
What Happens When User-Testing Results Go
Against the Grain?
However, are the results of
user-testing always gospel? We, at
FoxTranslate, a certified translation service ran into that exact issue. A few months ago, we redesigned our homepage. We felt that the redesign more effectively
conveyed our key service features and more professionally segmented the content.
Current
Homepage
|
Redesigned
Homepage
|
 |
 |
To validate our intuition, we
ran user-testing to compare the two sites, expecting our redesigned homepage to
be the clear champion. Unfortunately,
the results of user-testing suggested otherwise.
User-Testing Results
If you needed a document translated,
which page would you more likely use?
|
Common Themes for Picking the Current Page
|
||
|
“The
current homepage feels more credible and has a more professional look”
“The
current homepage feels more secure”
“The
current homepage is much easier to navigate and gather information”
“The
current homepage has a much better layout”
|
The results showed that users
clearly preferred the current homepage, meaning it might be time to head back
to the drawing board.
Call it stubbornness, but we
proceeded to test the redesign on our actual customers using Google website
optimizer. Google website optimizer tested the
effectiveness of our new homepage by sending half our customers to the old
homepage and half to the new homepage. From
there, website optimizer calculated conversion rate (# of transaction made /
site visitors) and revenue generated, allowing us to gauge which page lead to
more transactions.
Results of Current vs. Redesigned Homepage Test
Redesigned
Homepage Conversion Rate and
Transaction Size Improvement |
 |
Over the course of a month, our
new homepage generated 17% more revenue vs. the current homepage on similar customer
visits. 6% of the increase was due to
more converting visitors and 11% of the increase was due to people spending
more per transactions.
Our Learnings
So, why didn’t the actual
results align with the results from user testing? Sometimes, it’s hard to replicate your
customers – Regardless of the results; we are still advocates of user-testing. However, sometimes, it’s hard to find
everyday people to replicate your customers.
We’re not selling laundry detergent, so generic user panels can only
tell us so much because they don’t exactly know what your customers care about.
About the Author: Jason Thai is a marketing manager for FoxTranslate, specializing in document translation of business,
legal, immigration and academic documents in over 30 different languages.
Marketing Blog founder and top-selling author Michael Fleischner has been featured on The TODAY Show, ABC World News, and Bloomerg Radio. 
8 comments :
I got a lot out of this Blog post. I really stress doing market research. Your comment about "How will customers view the new feature.", rings true. I've been sending a SurveyMonkey by e-mail with marginal results.
What a lovely post.
What a lovely post.
A really nice topic picked and explain here by Jason Thai in this guest post.
Thanks,
As a business owner, recognizing your audience and appealing to their needs is essential if you hope to ultimately be successful. Knowing what your customers want and what appeals to them has the potential to make or break a sale and even your entire business. I agree completely with you that it is extremely important to do adequate research about your customers in order to best serve their needs and make the most our of your company.
Great Work dude.
You're blog is amazing! I have gathered so much information from this, specifically about video marketing http://www.mojovideomarketing.com/. I hope you can share more of your interesting blogs soon. I'll be waiting for it. Thanks.
Thanks for this informative post! Market research is really important before launching a certain product. Consumers point of views are important. :)
Post a Comment
Please note that we reject all posts that clearly are leaving a comment simply to acquire a back link. Only comment if you have something of value to share with other readers.
Thanks!