The Pros and Cons of Using an Online Testing Service

Usertesting image.png

Usertesting.com and UserZoom.com are quick and popular ways to test your website or app with actual users. These services have really caught fire and accomplished a great feat in opening the door to making user testing known and accessible to pretty much everyone, especially to teams that are totally new to user testing or may not have user research resources available to them. The service is extremely low cost (on UserTesting.com I tested with five users for just $165) and very easy to use: just go to the website, select from a few drop down menus, and in a couple of hours you have videos of users using your app or website. Sounds pretty awesome, right?

I thought so, too. So as a professional user researcher I thought I’d try it out myself so that I can advise my friends and clients about when the service may be most useful - and when they may want to try a different path towards understanding their users and how to best design a product for them.

Here’s what I found when I conducted a test with UserTesting.com:

Pros

  • It’s quick. Turn around for the testing is just a couple of hours.
  • Very cheap access to testers, which is often the most resource intense part of user testing.
  • I received very high quality feedback from the testers.
  • Clear video recordings of user interaction.
  • Quantitative data such as average time on task and NPS are collected and automatically analyzed.
  • You can reach out to testers for follow up. This can be a huge plus as marketing research agencies can charge up to $200 just to talk to one participant a single time.

Cons

  • You can only test shipped products. This is by far the most costly stage to test and update your product. There is no ability to provide confidentiality agreements or to facilitate tests yourself, therefore no chance at testing a concept any earlier.
  • There’s no summary. You will still spend the time to review the videos to understand the issues users faced and also what went right.
  • Any high level or strategic insights are missed.
  • There are no recommendations for what the problems are and how to fix them.
  • The test is totally ‘hands off’. Once you create your few tasks for the testers to do and submit the test, it is off and running. You have no opportunity to stop the participants, ask questions, or probe to better understand what a users is saying or responding to, while they are experiencing it.
  • It is unclear who the testers are and exactly how they qualified. The service says that they are ‘self selected’. 

Overall this service currently provides quick video data and some metrics (no summary) about a very specific task set in a fully developed user experience. Which is great, if the alternative is to gather no user input at all. But my best advice to people thinking about using this service is that while cheap, there are several very real cost tradeoffs to relying on an online testing service like UserTesting.com:

  1. It will cost you approximately 4x more to fix any issues you’ve found in a fully developed product than if you had tested your interaction design more strategically (meaning early and often - before it was fully developed).
  2. You will still need to leverage a user experience resource to go through the videos and help you determine the top issues and what is the best way to go about fixing them. This is a very real human resource cost not included in the $165.
  3. You risk including user participants who are NOT your target users. You have no control over the sample of your participants or test flow once you commit to doing the test. This is concerning, because the value of your test data is only as good as the testers you include.
  4. Lastly and most importantly, the process is largely reactive and not proactive. Teams will be more effective at creating user-centered experiences if they proactively seek ways to include user needs when building a UI from the ground up, rather than as a band-aid solution to fix problems.

In sum, I'm reminded of the old "Rich Dad, Poor Dad" mentality. Testing your fully developed experience using an online service such as UserTesting.com may provide a cheap and quick, near term fix to gathering user feedback. However, a more valuable investment would be to create a sustainable, ongoing, and strategic user-centered design process. I will continue to encourage my clients that instead of relying fully on small, cheap tests, they should focus their energies on building a more comprehensive yet resource efficient way to include their users’ needs as a central, driving force towards building - and not just testing - their user experience.