Users lie to you, out of love

I've been working on front-end for 25 years: the part of a website or app that users see and interact with.
In lofty terms: I work on the interface between humans and machines.
To explain it to Mom and Dad, whenever they asked “So… what exactly do you do?”, I came up with this: “I do my best to help normal people understand us computer folks, and vice versa.”
Simplistic but satisfying.
Sometimes I'd ask Mom or Dad what they thought of the websites I built and they, proud of their firstborn's hard work, would reward me with praise, and I was happy and satisfied with the answer. But I was wrong.

It's pretty obvious that Mom and Dad weren't telling me what they really thought, but what I wanted to hear. In other words, they were lying to me.
Without malice, but they were lying, because every mother thinks her baby is beautiful.

And even if you're not my mom or dad, you'll still prefer to give me the answer I expect: it's easier to tell me that, yes, everything's fine than to say you don't like it and take on the responsibility of criticizing me.

In technical terms, this is called cognitive bias: a prejudice or distortion that affects the answer, and it happens to everyone.

When we build a product, say a website, we want the opinion of people whose eyes are neutral and more objective than ours, since we've been staring at it so long we no longer notice its flaws. More concretely, we need to find out whether what we've built solves the problem it was made for, and the best way to find out is to ask the users it's meant for for feedback.

So, to sum up: we'd rather please others than criticize them and we're all swayed by cognitive biases of all kinds.

So how do you get an honest answer?
By asking the right questions.

When we do user research at Cantiere, we run into exactly this problem. That's why we've developed a working system that combines field experience with good practical ideas from a book called (fittingly) The Mom Test.
The book is aimed mainly at entrepreneurs and startup founders who want to validate their business idea before building it, but the method adapts easily to qualitative interviews for existing products or to gathering other kinds of feedback.

Our system boils down to these guidelines:

• invite interviewees to talk about themselves and their experience with the product, instead of asking for a flat opinion;
• the future is hard to judge: ask specific questions about things the user has already experienced, instead of asking for guesses or opinions about things that don't exist.

This works very well in live qualitative interviews, but it also applies to online surveys.

Here's a quick example: we've prepared a short survey with 5 questions, some good and some bad. Click the big button below to rate them.

Rate the questions

Made your choice? Great, here are the right answers.

“Do you think this website has nice graphics?”
Bad question: interviewees will resist saying they don't like it, and it's a question about an opinion, which at best gets us a simple yes/no that's no use to us.

“How much would you be willing to pay for this service we're creating?”
Bad question: if they're not interested in the service, they'll be reluctant to say they won't pay anything; if they are, they'll have to guess about the future without knowing themselves how much they'd really pay. Not to mention all the useless “as little as possible” answers a question like this collects.

“How did it go the last time you booked a hotel through this website?”
Good question: here interviewees can't just please us, because they'll talk about themselves and their experience, and we can understand the strengths and weaknesses of the booking system;

“Would you use a product that does X?”
Bad question: again, we're talking about guesses about the future, and the answer is a simple yes/no;

“Tell me about your workflow.”
Good question: no people-pleasing bias, and interviewees will happily walk us through their process, so we can see how to improve it or where we went wrong when we created the flow we're evaluating.

If we give users room to express themselves freely, they'll give us lots of valuable information, and we'll get more honest answers, less tainted by bias, on which to build stronger, more reliable hypotheses and assessments.

Sometimes interviewees even spontaneously share very interesting things we hadn't thought of while preparing the survey, giving us more useful insights that would never have come up with the wrong questions. In fact, the most important guideline of all is:

• talk less and listen more.

Finally, here are some of the questions from a survey we prepared for a client, a school, to assess its students' problems and wishes.
Two of us conducted the interviews, talking to one student at a time, in the most informal and casual tone possible to give interviewees plenty of room to tell their stories.
The questions have been anonymized for privacy reasons.

1. Why did you choose this school?
2. How are you finding it?
3. How do you get along with the other students?
4. How do you get along with the teachers?
5. How do you communicate with them?
6. Tell me about the last difficulty or problem you faced
7. When did you last use the school website?
8. To do what?
9. What do you expect to find in the new-product-we're-preparing?
10. Is there anything you'd like to add?

Following this style, you can expect detailed answers full of concrete (and sometimes unexpected) facts on which to build strategies or make decisions about the product you're building.

Happy user research!