UX Events: Usability Testing Metrics

Usability testing is a user-centered approach to evaluate a digital presence by testing it on users. The ultimate goal is to identify any usability and design issues. In usability testing, researchers will create a real situation or scenario and guide the users to complete some tasks over the prototype or actual products. They will observe the participants while gathering feedback to further understand the users' perceptions. Rather than listening to the users describe their experience, usability testing aims to observe the behavior of users. In recent years, usability testing has emerged as one of the most favored research methods by stakeholders. There are two main types of usability testing: Moderated Usability Testing Moderated usability testing is conducted in the setting of a lab, contextual inquiry or hallway testing. On the other hand, remote moderated usability testing is also gaining popularity to save time and cost. Read more about remote qualitative research for UX. Unmoderated Usability Testing Unmoderated usability testing is always done online. Researchers usually go for unmoderated usability testing when a large quantity is required. Respondents have the luxury of time to complete the tasks at their home over their computer or mobile devices. Benefits of Usability Testing It measures behavior, not preference. Most of the time, users are bad at articulating what they want. By studying their behavior, you can understand what suits their goals and motivations. The reliability of insights from ‘we saw someone doing this’ is far beyond that of ‘we heard someone mentioning this’ from a normal focus group session. It gathers both quantitative and qualitative data in terms of effectiveness, efficiency, and satisfaction. You will be able to explore the area of improvement in functionality for your digital presence. Creating a Testing Plan or Proposal You don’t wake up one day and say “hey, let’s do a usability testing on our products”. Usability testing is not something random. While widely perceived as a qualitative approach, with the appropriate approach and consistency, usability testing can be quantified. It is more than just handing an application to the participants and recording the observation. Firstly, create a testing plan or proposal. A comprehensive and methodological testing plan ensures a systematic process and a consistent data set. These make it easier to convince the management for the feasibility of the study. A good testing plan includes research objective, suitable subjects or personas for the objective, a set of guidelines to ensure the consistency of this study with past studies and future studies, tasks and the corresponding measurements and metrics scripts A testing plan seems ridiculously tedious for the first time but you can use this testing plan as a foundation to create plans or proposals for all types of research in future, not just limited to usability testing. Turning Research Objective into Tasks You need to convert the research objective into something for the participants to do during usability testing. You can call it tasks. Make the tasks as realistic as possible. For example, you are planning to measure the usability of the version update of your travel accommodation booking site. You can phrase the task as below: You are planning a trip to celebrate Christmas in London from December 25 to January 3 with your spouse. Go to Expedia.com and book your accommodation with budget of less than 300 pounds per night. Try to make the tasks actionable without giving hints or step-by-step guides to the participants. This ensures that the participants have the freedom to complete the tasks based on their habits and preference. In the process, you will get to see firsthand the functionality issues of your site. Identifying the Metrics to Measure The Tasks Depending on the tasks given during usability testing, you can propose different metrics to measure the tasks. These metrics are typically used in measuring the effectiveness of completing a search over e-commerce platforms: Success rate Time taken to complete a search Total page views or mouse clicks System Usability Scores (SUS Scores) Satisfaction rate However, even with the results and metrics, it is not meaningful without a benchmark to compare against. For example, if you are measuring the usability of your e-commerce site’s proposed update in terms of ease to complete a product search, you may compare the task results against the results of existing design or the results of competitors. Compare Against Existing Design It is important in usability testing to have a set of procedures to ensure consistency across all the studies. With such consistency in research, results from previous usability testing could be used as a benchmark to determine the usability of any new design or upgrade. The example below illustrates the results comparison of the current published version (version 2.3) against the proposed version update (version 3.0): Version 2.3 Version 3.0 Task 1 Average Success Rate 60% 70% Average Time Taken 130s 120s Average Total Page Views 7 7 Task 2 Average Success Rate 50% 60% Average Time Taken 50s 45s Average Total Page Views 3 3 Average SUS Score 70 75 Compare against Competitors While you are busy improving the usability of your product or website, your competitors are doing the same and might be better at it than you. That’s why you should frequently conduct usability testing on your competitors’ products and benchmark yourself against the results. It is always important to be aware of the latest technology and how your competitors are performing. One might argue that the effort spent researching competitors would serve a better purpose on more in-house research and development activities. However, it may come as a potential loss for the company for failing to follow the next big trend and a potential revenue for the competitor. The example below illustrates the results comparison of proposed version update (version 3.0) against Competitor A and Competitor B: A B Version 3.0 Task 1 Average Success Rate 70% 75% 70% Average Time Taken 130s 110s 120s Average Total Page Views 7 4 7 Task 2 Average Success Rate 50% 65% 60% Average Time Taken 50s 40s 45s Average Total Page Views 4 4 3 Average SUS Score 70 85 75 While Version 3.0 outperforms Competitor A, it is lagging behind Competitor B in terms of key metrics. During usability testing, researchers can further explore the functionality improvement needed for Version 3.0. Writing the Scripts Preparing a script in advance will make the research more organized. Every participant will go through the same process without the difference in research flow, contents, and choice of wordings. Some researchers prefer the research to be more casual or spontaneous and reading out from a script appears awkward to them. Well, they just need to practice the script one or two times in advance and it will be awkward no more. Practice makes perfect. It is strongly advised to rehearse the whole research in advance such that you can get a gauge of the time needed and identify any potential issues that might occur during the day. Refer to the sample script below: Introduction Hello, John, I am Eric, the researcher for today. I am going to give you a briefing on the session for today. Today, you will be asked to perform some tasks on website A. We are interested to know how is it like when an actual user is using it. Rest assured that we are testing the website, not you. There's nothing wrong here, please give us your honest opinions, be it negative feedback. We like negative feedback as it can help us to improve the website. Please think aloud (speak your mind) as you work on the tasks. You have probably noticed the microphone and the camera. Yes. We would like to record the whole session with your permission. It will help us in improving the website and it will only be seen by researchers working on the project. Will you be good with it? If you are good with it, before we start, do you have any questions for me? Background Information Prepare a form with background information for the participants to fill up before the session instead! It will save you more time during the usability testing. Usability Testing Go to Expedia.com. Firstly, without clicking anything, tell me what do you think of the website and where would you first click. You are planning a trip to celebrate Christmas in London from December 25 to January 3 with your spouse. Now, book your accommodation with a budget of fewer than 300 pounds per night. Try to think out loud as you proceed. How Many Participants Is Enough for Usability Testing? For a normal usability testing to identify issues and potential improvement for a website, experienced UX professional will tell you that five participants are more than enough. Anything beyond five will be near saturation. Saturation is a phenomenon which occurs when subsequent usability tests do not provide many additional insights. Unlike quantitative research where you determine the sample size needed with confidence level and error margin, qualitative research typically requires less sample size. For your study results to produce statistical significance, plan to evaluate with 10 to 12 participants per condition. Recruiting Participants Firstly, you will need to decide whether to recruit on your own or outsource the participant recruitment. If you choose to do it on your own, you will need to find the right participants, decide the optimal participant incentives and manage the participants to prevent no-shows. Refer to the SOP for participant recruitment here. Setting-up A Research Lab Some companies like to have a study conducted at their own facility to save cost. However, for a brand-blind or sponsor-blind study, it is more appropriate to choose an external facility. Regardless of whether it is an internal or external facility, you will need to prepare to tackle the hassle of setting-up the environment. Refer to the SOP for set up of a research lab here. Conclusion A professional usability testing requires a good planning and significant effort. As a qualitative research requires more effort than a quantitative research in planning, any potential error is an opportunity cost. Go through the following checklist one or two days before the usability testing: Have you conducted a dry run with your colleagues? Have you familiarized yourself with the script for tomorrow? Is the recording facility ready? Have you or your agency called the participants to prevent no-shows? With these, you are all set for your usability testing.

UX Events: Usability Testing Metrics

Usability Testing: What is usability testing & its best practices

Latest UX Job Openings

Research Design Prototyper

Quantitative Staff UX Researcher

Principal User Researcher

Senior UX Design Reviewer

Senior Director of Design, Yahoo Sports

Sr. UX Researcher

Upcoming UX Events

iF Design Award 2024 (Berlin, Germany)

CHI 2024 (Online / Honolulu, Hawaii, United States)

UX360 Research Europe Summit 2024 (Berlin, Germany)

UXLx: User Experience Lisbon 2024 (Lisbon, Portugal)

From Business to Buttons 2024 (Stockholm, Sweden)

Worldwide Developers Conference 2024 (Online)

Latest UX Articles

Top 5 Content Strategy Trends 2024 / 2025: AI and beyond

Top 6 Web Design Trends 2024: Colors, Content, Speed

2024: Top UI UX Trends you shouldn’t overlook

Top 5 Content Strategy Trends 2023

Top 8 Web Design Trends 2023

What is Penpot? What is the hype about Penpot?

Sign up for our Newsletter