Why You Should Believe User Tests That Involve Only 7-9 Users

I have been conducting User Experience Research as a professional for the past twelve years, and I have learned many things about how to do it correctly as well as -  although I am sometimes embarrassed to admit - incorrectly. I have been known to use trial and error to see what works with users, to explore and determine how I can get authentic responses from them when testing out new experiences, products, or applications.

 

I have worked, reworked, stretched, and pulled just about every known user research method. These include focus groups, usability testing, guerrilla ‘intercepts’ at coffee shops, Craigslist surveys, and all the other usual methods, manipulated to meet my clients’ constrained resources and timelines. I have experimented with limits of scientific inquiry, and Wizard of Oz’d just about every technological interaction from using slight of hand to hidden foot pedals, all to ensure that the user has a voice in the early stages of product and application design. I can tell you from personal experience it is incredibly rewarding while often times quite exhausting. 

Yet beyond all of this physical and intellectual maneuvering, far and away the most persistent challenge I seem to face as a UX Research practitioner is the regular dismissal of qualitative studies involving a small sample of participants. I am referring to product developers and team managers who frequently say to me in response to my qualitative, ‘quick turn around’ user studies - “But how can you claim to find patterns when you only talked to 7 users?” or “We can’t use that recommendation because it’s coming from a small qualitative study.”

Don’t get me wrong, I totally get this push back. I completely understand the reluctance of analytical-minded tech experts to put considerable investment into feedback coming from only 7-9 people, when they want their product to appeal to millions of customers. I want them to succeed with millions of customers, too. I usually respond to these kinds of questions with an understanding nod and a deep breath before explaining that qualitative data, gathered through a smaller sample, provides a unique kind of knowledge about users. Knowledge that is critical to the success of your product. Here’s why.

First I will admit that in some ways, they are right. Surveying 7-9 users simply can’t scale enough to enable the research to answer questions about exactly how many people would experience a certain problem, or how large of a market a product has. But it does allow me the time and resources to dive deep into a particular experiences, contexts, and emotional connections with individual users. With this kind of data I can tell the team why and how their products are having an impact - good and bad. Data they can actually use to improve a product. Realistically, if I took the same amount of time and resources and paneled a large survey with say, 950 participants, I would be able ask them all what they are doing via closed, multiple choice responses. I would not be able to ask them why they do what they do. I would not be able to dive deep into the richer information about the respondents’ personal experiences, contexts, connections, associations, and emotions which I can by instead taking the time to interview just 7-9 users in their homes.

More specifically, with a large quantitative study I could not see the users’ smiles or frowns, watch users accidentally launch Siri from their iPhone when trying to open an app, see their children tug on them to show them how use a game on their iPad, or witness users struggling to find a TV program on Hulu. These are the rich, qualitative data that product teams absolutely need to understand in order to design products and applications that are compelling and fit seamlessly into their customers’ lives. 

As Apple has shown us, building great products that create an emotional connection with users is key to success. Only small-size qualitative research studies can enable us to collect the rich data and understanding about users’ authentic emotions, and gather why they feel the way they do about certain experiences and products. Think about it - a pictures tells a thousand words, right? Well guess what - a picture is qualitative data. Let’s compare:

Here are some numerical (quantitative) data:

The share of adults in the United States who own tablet computers nearly doubled from 10% to 19% between mid-December and early January and the same surge in growth also applied to e-book readers, which also jumped from 10% to 19% over the same time period.

source: Lee Raine, “Tablet and E-book reader Ownership Nearly Double Over the Holiday Gift-Giving Period” (2011). Pew Research Center’s Internet & American Life Project.

This is pretty good stuff. I now have a good idea of an increase in tablet and E-reader users out there. The researcher had a large sample, and I trust his methods. 

Now, some photographic (qualitative) data:

Transient

This is a picture of one excited customer receiving her newly purchased E-reader/tablet device. 

Would you dismiss this evidence in this photo, because it is only one user? Probably not. She looks pretty happy. 

This user is excited, and I can tell you why (okay, full disclosure- that person happens to be me. I tend to keep my participants’ personal information, such as pictures of them, confidential but I assure you what is captured here is a real moment of happiness when I received my first E-reader device). If this person was a participant in my qualitative study, I would be standing in front of her capturing her smile, and asking her why she is smiling. I would interview her at home while she received this device, and I would learn that she is very excited about reading The Hunger Games, which she knows is preloaded on the device she just received. I could learn that she thinks she’s going to get to finish those books as soon she opens the box (she does - success), and that she was so excited about her E-reader working immediately that she showed it to everyone around her. In sum, I can tell you a lot about how this user feels about her new device and why because I am able to be there to observe and carefully interview her about and during her experience. All of these findings are possible only by conducting a smaller sample, qualitative study.

Okay, so what about patterns, you ask. How can you claim to find patterns, when you only talked to 7 users?  Let’s look at a qualitative sample of 7 users:

Transient

Let’s say that 5 users expressed excitement upon receiving their new E-reader device and 2 did not express any particular excitement. Although it would be misleading of me to generalize 7 users’ experiences to the total population of possible E-reader users (for example, saying that 5/7, or 71% of all e-reader customers are excited when they first receive their device), one certainly can conclude from the qualitative data (user photos) that the majority of the small sample exhibited pure excitement upon receiving their device for one reason or another. This, and why exactly they are excited, is kind of data I usually get through small sample qualitative research. 

Any number of customers experiencing excitement with your product is an important pattern to pay attention to. You may want learn about and build upon that excitement, or even more importantly, understand and fix any disappointment. These are the kinds of patterns in emotion and experiences that occur after observing 7 to 9 users in a qualitative study. In my experience, any observable patterns at this point may not represent ALL possible experiences with your product, but are likely to be relevant enough to repeat with continued researchJakob Nielsen would likely agree with me

Once you are open to it, qualitative data with just a few users can be powerful. It’s the kind data that can bring you closer to your users and move people and teams into action. It can inspire and direct you to build better products. I encourage you to believe in small sample qualitative data to your company’s and users’ benefit.