A Practical Guide to “Light” Usability Testing — Part 2

We finally get to the most important part. Let's step into the single hour dedicated to each participant and break it down into phases.

We are now in the Test Room (not sure what the Test Room is? Read part one of the guide). Our participant is sitting at the computer, and we are next to them with our notepad and a few extra sheets we'll explain shortly.

Image Source: Stippen Consulting

Everything is set up, and you have already checked the audio connection and screen sharing with the Observation Room.

Here's how the test unfolds:

1. Introduction (4 min)
2. Who the participant is (2 min)
3. Homepage tour (3 min)
4. Tasks (35 min)
5. Our questions (5 min)
6. Participant's questions (5 min)
7. Break (15 min)

The participant has taken a seat and we're starting the session.

First, we read the text below. It's best to stick to the script and not improvise, because some words can have multiple meanings and be misunderstood, especially at the start.

Here you'll find the consent form in English, under “Recording Consent Form”.

Before starting the tasks, you need to ask the participant a few quick questions. This gets them warmed up and talking (soon they'll have to think out loud, the Thinking Aloud Protocol), shows them that you're fully listening, and helps you understand their level of technical knowledge of the internet and how close they are to the audience you picture for the site. It also gives you a rough user profile of the person you're interviewing.

OK, before we go to the site, I'd like to ask you a few quick questions.

1. What do you do for a living? What do you do all day?
2. Roughly how many hours a week would you say you spend online, including web browsing and email, both at home and at work?
3. As a percentage, how much of that time is browsing versus email, or vice versa?
4. What kinds of sites do you usually visit?
5. Do you have any favorite sites?

Let's begin. The homepage is the first thing we tackle.

Here's what to say:

This helps you understand whether the messaging on your site's main page clearly reflects the service the site offers overall.

Does the cover of our product tell people what the product is about? What does an average human being think?

It's important to stick to this: we're asking the participant to figure out what the site is about starting from the homepage; we don't want their opinion on the homepage's graphics or style.

In this phase the user can only scroll the page, without clicking anywhere.

We've reached the heart of our test.

First, give the participant a copy of the first scenario and read it aloud yourself.

Here's what to say:

Once the participant starts the task, try not to interrupt; at most, remind them to keep thinking out loud if they tend to stop.

While they navigate, watch in silence and jot down any notes that come to mind. The same goes for the team in the Observation Room.

When is it time to move on to the next scenario? Obviously when the participant has completed the task correctly, but also when the process turns into a dead end and the participant gets too frustrated, and when you realize you're short on time and want to explore the remaining tasks properly.

Follow this format for all the scenarios, and always keep an eye on the clock.

Additional questions will come to mind (for example: what matters most to you in this task? What information do you want before...? What didn't you understand about that scenario? Didn't you know this technical term? etc.).

Write them down; you'll ask them in the next phase.

Finish all the scenarios, and if there's no time for some of them, unfortunately you'll skip them and move on (with the next participant you'll have a better sense of timing).

In this phase you ask any questions you've noted down.

Check in with the Observation Room and see whether your colleagues have any questions. Manage your time your own way: this phase can be very long or very short, so handle it according to your circumstances (Steve Krug recommends five minutes).

Here's what to say:

We're done. Thank the participant and ask whether they have any final comments. Take all their style suggestions with a grain of salt, and focus on their difficulties and misunderstandings.

There will be a ten- to fifteen-minute break, during which you will:

1. Save the audio
2. Save the screen recording
3. Clear the browser history and cache
4. Tidy up your notes
5. Write down anything else that comes to mind

After that, you open the door to the next participant.

Once the three tests are done, we move on to the debriefing.

Good, we've finished the three test sessions. The team has already had lunch, it's now 3:00 p.m., and we're about to start the debriefing, which should last one hour.

We need to get two things out of this meeting:

1. A list of the most serious usability problems
2. A list of fixes decided on the basis of those problems

The debriefing really should take place on the same day, so the experience with the participants is still fresh in everyone's mind.

We recommend letting only members of the Observation Team into the debriefing. This avoids “religious debates” (as they're called) based on personal opinions.

A guiding principle for the meeting is:

There will be plenty of problems. Many people will try to share their opinions, which often go beyond what was actually observed that morning.

Many will blow up problems that are actually minor, and that have more to do with their own stylistic vision or a long-standing pet peeve than with an objective assessment of that problem's consequences.

Let's remember that we're gathering the users' experience, not the team's.

You need to be able to rank the problems objectively by how “dangerous” they are.

In UX design you have to learn to be objective and set aside how expert you feel about the internet or how good you are at product vision. You need to be realistic and scientific.

That said, whoever leads the debriefing needs to keep the atmosphere calm and collaborative, and make sure the above doesn't become a workaholic commandment. Usability tests are always rather delicate, because they call into question the hard work of many people, so they require seriousness but also tact and understanding.

Here's roughly how a debriefing can go, in order:

Start like this:

Then, in order:

1. Ask everyone to review the list of problems they noted during this morning's tests, and to pick the three they consider most urgent.
2. Ask each person to read out their three problems one at a time.
3. Write them all on a whiteboard.
4. Once you have the full list in front of you, start marking (with checkmarks) the ten problems you think are most urgent, then ask the team whether they agree with your choices. Adjust if needed.
5. Reorder the list based on the priorities you discussed for each task. Write (1) next to the biggest problem, and so on.
6. Go back through the list from the first item to the tenth, and for each one discuss the fixes with the team.
7. Write the list of fixes.

Thank everyone and close the session.

Gather all the material and send everyone a recap email. You'll rework the material based on your experience and circumstances. There's no need for big reports with slide decks: within your team you'll know what's actually useful and how to present it. It's pointless to spend hours on reports when usability tests are run this often.

I know it isn't easy to pitch something like this to your team and have everyone spend a whole morning watching a screen. If you've read this far, you're clearly the right person to do it. Because “simple testing” doesn't take great skills; it takes understanding why running these tests matters.

Let's ask ourselves something. What's the point of building products for users we don't know? I'd rather not call them users; I want to stress that they're human beings. What I really mean is that there are people in the world nobly building technology for humanity today, and Italy shouldn't fall behind. We need to get to know these human beings. Because sometimes we're building our own little personal toys, our own whims, without even realizing it.

Let's ask ourselves: is my product really useful? For whom? Have I ever watched up close a human being using my product, whose life is genuinely made simpler by it?

If I've never met an end user, what am I building on? Google Analytics? Surveys? Maybe I'm spending months of my life building my idea of a product. On what idealistic basis should mine be a useful product? We need to weigh facts, not ideas. Google Analytics, Kiss Metrics and Mix Panel give us numbers. But where are the people? Are we really useful to humanity with what we build every day?

By running usability tests, we get to know the people without whom our product, our office and our working hours wouldn't exist. Personally, I love how many of the tools I use make my life easier. For example, I'm loving using Sketch instead of Photoshop, simulating prototypes in InVision, running an e-commerce store on Shopify and managing mailing lists in Mailchimp; and I wish the founders of Sketch, Photoshop, InVision and Shopify were right here next to me now.

One day we'll email a long-time user of our platform, give them premium subscriptions and some swag, and interview them remotely using the techniques we've talked about today. It's not just our wireframes and prototypes that are user-centered, but our mindset too, always. Our mission is user-centered, human-centered, humanity-centered. No ideas or abstractions: we interview humanity and work for humanity.

Without a doubt, “simple testing” is also about connecting with the heart of our projects: humanity.