How NASA can help you build a better app

Have you ever wondered if your mobile app is too much work for users? Or if your user experience feels too ‘heavy’? Or if using a specific app while carrying a device is too demanding or frustrating? Well good news: there’s a little known but very handy research tool that you can use to help answer these questions with confidence.

It’s called the NASA Task–Load Index or NASA-TLX for short. And because this tool was developed by NASA researchers in the eighties to help to with human factors in designing aviation systems, it includes measuring dimensions of mental and physical demands on users – quite pertinent to today’s physical and mobile interfaces. Over the past couple of years I’ve been finding ways to use NASA-TLX to help improve modern-day mobile app user experiences.

What exactly is the ‘NASA-TLX’?

The NASA-TLX is basically is a set of subjective ratings and scales which measure a user’s ‘cognitive workload’ while completing a task or set of tasks. If you are familiar with simple ‘please rate between 1 and 7’ Likert-style ratings or even the classic System Usability Scale (SUS), you should find the tool to be relatively approachable and easy to use.

What can the NASA-TLX tell me?

The NASA-TLX will give you a simple, easy to read number (or quantitative score, if you will) for the overall ‘workload’ a user feels while completing a task or set of tasks on a system. It will also break out for you the impact of various contributors to the load that users experience when operating a system or UI, including the following:

  • Mental demand
  • Physical demand
  • Temporal demand
  • One's own performance
  • Effort
  • Frustration

Here is what the ratings sheet looks like in the simple paper-and-pencil version.

NASA-TLX rating scales

NASA-TLX rating scales

The NASA-TLX is also unique in that it allows for respondents to determine which of these factors most impacted them during completion of the task. This form of weighting the factors enables your users to share with you not only how much they felt of each factor, but also how important a given factor was for them relative to other factors while completing the task.

Example of load factor ‘weighting’ in the NASA-TLX

Example of load factor ‘weighting’ in the NASA-TLX

By including not only the ratings but subjective weighting of the rankings, the NASA-TLX provides a much more detailed and clear understanding of what kinds of cognitive and physical demands your users feel while using your app versus regular rating scales. Of course, it will be up to you, the experimenter/researcher, to observe use and also ask users why they rated the factors what they did, enabling you to translate the scores back to a given feature or UI element.

Issues and limitations

As with any data collection tool, there are some limitations with the NASA-TLX. For instance, the rating questions could use some rewording and updating. Several of the descriptions are unclear and I’ve found myself having to clarify to the respondent multiple times what I am asking them to consider, such as the difference between ‘Performance’ and ‘Effort’ factors.

In addition, situational or multitasking factors should be included into the overall consideration of ux workload. For mobile experiences I find this to be increasingly important. The load on the user for texting while driving, for example, comes to mind. It is not clear if or how the NASA-TLX attempts to incorporate various contextual factors or distractors into the ratings.

Lastly, other factors including emotional and motivational demands on accomplishing a task are missing from the tool and should be included in any overall workload consideration. If I have a strong affinity to a device such as my iPhone (for example), does that weigh into how important I feel the physical demand is on me while using it? Until these factors become integrated into an equally easy to use toolkit, I’ll just augment those metrics on top of the NASA-TLX when doing these types of investigations.

Try it out for yourself and get the data behind your UX

Like the classic system usability scale (SUS), you can use the NASA-TLX scores not only for immediate evaluation of your ux but also as a clear benchmark for future testing and iteration. What I personally find so uniquely useful and cool about this tool is that instead of system usability – a very system-centered metric – NASA-TLX is collecting a deeper understanding of how the user feels while interacting with the system and what factors are driving those feelings. This type of deeper understanding of the user experience is critical in building a desirable and competitive application.

In my next blog post I'll provide a case study using the NASA-TLX to evaluate a couple of mobile productivity apps, Asana and Trello.

For full NASA-TLX details and instructions sheet, go here.