User Research: What we love talking about, but rarely do justice to

Issue 35

4,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. If you are new here, receive the newsletter weekly in your email by subscribing 👇

---

Happy New Year 2021,

Hope this year brings happiness and well-being to humanity :)

I published this post on the substack on 27th Dec. I was also trying to shift the newsletter URL from https://deepaksingh.substack.com/ to https://www.growth-catalyst.in/. And due to some technical issues during the process, I could not send emails after shifting. So this post couldn’t go through email to your Inbox.

I shared the post on social media channels like Twitter and LinkedIn though and some of you may have read it. The issue with email got resolved this week only. So sending it this week for most of you who haven’t read it yet :)

From the next week, here is the schedule of the posts in the next couple of months, spaced at 2 weeks each:

• User Psychology - Concepts to Convert Users with Real Examples [24th Jan]

• User Psychology - Concepts to Retain Users with Real Examples [7th Feb]

• Masterclass to Gamification - Part I [21st Feb]

• Masterclass to Gamification - Part II [7th March]

• Masterclass to Gamification - Part III [21st March]

Off to the post,

---

Airbnb recently did an IPO of $100 Bn. One of the key stories from the early days of Airbnb is what Paul Graham wrote in his essay “Do Things That Don’t Scale”. In this essay, Paul Graham advises startups to do things that don’t scale while building the startup initially. And the most common unscalable thing founders have to do at the start is to recruit users manually. You can't wait for users to come to you. You have to go out and get them. He wrote in the essay around AirBnB,

You'll be doing different things when you're acquiring users a thousand at a time, and growth has to slow down eventually. But if the market exists you can usually start by recruiting users manually and then gradually switch to less manual methods. 

Airbnb is a classic example of this technique. Marketplaces are so hard to get rolling that you should expect to take heroic measures at first. In Airbnb's case, these consisted of going door to door in New York, recruiting new users and helping existing ones improve their listings. When I remember the Airbnbs during YC, I picture them with rolly bags, because when they showed up for tuesday dinners they'd always just flown back from somewhere.

Founders don’t have a choice when it comes to user research in the early days of building and growing the product. The market and insights are super important to the founders to build a great product. And once founders set themselves down that path, no one can know about the users and the market better than them. Over time, it just becomes a habit and a superpower.

Is there a scalable way to do user research? Companies hire research teams over time who go to the market to bring back insights. However, product and growth teams still have to invest a lot of time going through raw recordings to get meaningful insights. In my opinion, user research never scales really. It is one of those unscalable things founders, PMs, and growth teams have to keep doing day-in, day-out. The market changes with such swiftness and without mercy that companies get obsolete every other day. To stay relevant, we have to invest a lot of time talking to consumers, going through market reports ourselves. Insights are hard to delegate/outsource :)

Hopefully, I have made an interesting case for why PMs and growth teams should care about user research🤔 In this post, we are going to cover the most important aspects of user research. But first a word of caution!

The world of user research is vast and might make you feel like an outsider. It made me feel so. Just look at this chart to understand why I am saying so :) Don’t spend more than 20 seconds.

The chart gives anxiety to a lot of folks I know. But as steve Krug wrote in his book,

“Fortunately, much of what I do (research) is just common sense, and anyone with some interest can learn to do it.”

We are going to cover what’s vital for product and growth teams in this team in simple terms. So this essay is helpful to anyone who is remotely interested in user research.

More specifically, we are going to cover four things

1. Market research versus user research

2. Why do user research?

3. 7 factors that affect the user research outcomes

4. What kind of user research is useful in different stages of product development?

5. Where and how do you start? (for smaller teams)

Market research versus user research

It is also important to understand what user research isn’t. People often confuse market research with user research.

Market research is useful in bringing business insights into market needs, size, competition, and pricing.  It uses quantitative methods and has a heavy focus on numbers. The researchers look at large sample sets to understand the average age of potential users, their income level, and other general characteristics.  Market research is more general than user research.

User research is mostly qualitative in nature. You can use data from your product or users to validate/invalidate the insights. It operates on a small subset of people to learn about who they are and what they want. We will be focussing on user research in this essay leaving market research for some other day.

The importance of user research

I have already made a strong case for user research. Let’s get into more details on where all it helps as some of these use cases you may not be familiar with

• Exploring the problem, understand why and not just what the problem is

• Getting behaviors of users and not opinions

• Validating assumptions and remove biases

• Discovering usability problems in product

• Creating personas by classifying different kinds of users you meet. Changing design, copy, experience according to these personas.

• Creating a shared understanding of user personas and their problems within internal teams like design, product, business, and engineering.

7 factors that affect the user research outcomes

A lot of big decisions are taken based on user research. But like any method, user research also has its own set of limitations. The decisions affected by these limitations wouldn’t turn out to be good decisions.

Let me put another point to explain this even further. All companies know that user research is important. They also have user research teams, but even then they fail to uncover customer problems and build products that don’t add much value to customers. Why is that? Because of the process and limitations of research methods matters. The quality of decisions made is as good as the limitations.

Hence it’s important to understand and remove these limitations wherever possible. There are 7 limitations —

1. Stated vs actual preference - In the last essay, we mentioned the ask-observe framework.

There is a key difference between asking and observing a customer. Asking provides you the stated preference of the customer, i.e. what people talk about/state. A lot of times, these stated preferences are different from actual preferences.

The actual preferences are revealed by observing what the customers actually do. They are also known as revealed preferences as they get revealed as you observe customers.

You may have seen multiple instances of stated versus revealed preferences in your day-to-day life. Like someone who says they like you a lot, but then go ahead telling lies about you to others.

Stated preferences are great for marketing products. But revealed preferences are the key to building products. What people say is more important in the marketing department. What people do is very important in product departments.

2. Environment - The environment in which the research is conducted is pretty important. People feel more comfortable in their own homes as compared to labs. Usually, the research should be conducted as naturally as possible.

3. Incentives — Almost all studies offer incentives for people for studies. It is important to be careful in places this can backfire.

Suppose you are designing a survey where you are paying people to complete a survey. In order to save time, people might start ticking random options without reading options.

Google designed surveys beautifully by creating a survey where they insert a fact-based question in between. The question can appear randomly and check whether the user is honestly answering the question. What a beautiful way to solve the problem with incentives, though it doesn’t solve it for all use cases.

4. Sample size — Depending on the user research method, the sample size of the TG varies. A usability study requires 6-8 participants. After 6-8 participants of a particular persona, the usability issues start getting repeatable. A survey requires >500 participants to contain error rates within 5%.

5. User’s relationship with the interviewer — Your Mom, family, and friends will often tell your idea is a good one as they want to cheer you up. Take as many unknown users in your research if possible. From the book Mom’s test,

They say you shouldn't ask your mom whether your business is a good idea, because she loves you and will lie to you. This is technically true, but it misses the point. You shouldn't ask anyone if your business is a good idea. It's a bad question and everyone will lie to you at least a little . As a matter of fact, it's not their responsibility to tell you the truth. It's your responsibility to find it and it's worth doing right .

6. Extent of context transfer — A picture is worth a thousand words. A working product is worth a thousand pictures. Does the user fully understand the context of your question when you ask them? If not, you may get the wrong answer.

Henry Ford is famous for the quote “If I had asked people what they wanted, they would have said faster horses.” Unless you show people what a car is and it can take them faster, you may not get the right assessment.

7. The researcher — The last and the most important limitation is the researcher himself/herself. The insights of the researcher are limited by the researcher’s ability to improvise in interviews, connecting with users and their needs, connecting with business/company and their needs. For this reason, sometimes founders and PMs are the best researchers. They are the ones working on it day-in, day-out. How can someone working for a couple of days, conducting some interviews get better insights than them?

If you go through solving all the limitations of the user research, you will quickly realize that it’s not possible to scalably solve these as there are a lot of nuances.

Stages of Product Development and User Research

As Lao Tzu says, “To attain knowledge, add things every day. To attain wisdom, remove things every day”

The wisdom lies in doing the research on a regular basis, rather than learning all these fancy terms from fancy books.

The best products get built, not by adding a lot of features, but by removing the clutter and solving a few core needs of the users. Take Apple, Netflix, Twitter, Uber as few examples. But saying no to features requires conviction, and the conviction gets stronger as you spend more time with consumers.

To do user research throughout product development, you have to understand different kinds of user research.

1. Usability Testing — Useful at all stages of the product. In usability testing, you let the users use a product and record the session to analyze where they struggle while using the product and what they love about the product. Usability tests should be done on every product. It also becomes useful when done on high fidelity prototypes before getting the product into development. You can also let users use your competitor’s products to figure out what aspects they have done well with and come easy to users.

   

   Usability testing can be done moderated or unmoderated. In a moderated study, the facilitator can gently nudge a quiet participant to share more about what he's doing. In an unmoderated study, you can ask a user to think aloud, but there is no one there to remind her if she doesn't do it.

   Nowadays, even remote usability testing is possible where usability testing is done via screen-sharing. Unmoderated studies can be done remotely through platforms like usertesting.com.

   Useful for: All stages, especially useful for prototypes, competitive analysis

2. User Interviews — Contextual interviews are highly effective in the exploratory phases of the product. There are two cases in particular. If you are trying to build a new product, contextual interviews help you connect with the users. If you already have a product and are struggling to figure out why something isn’t working out, user interviews are highly effective. You can do these interviews in person, video calls, or over voice calls.

   

   Useful for: Building a new product, figuring out why something is/isn’t working out

3. Survey and Questionnaires — Once you have done user interviews, you can quantify whether the problems exist with the bigger base by doing surveys/questionnaires. It is usually a bad idea to directly start with a survey. Qualitative methods are much better suited for answering questions about why or how to fix a problem, whereas quantitative methods do a much better job answering how many and how much types of questions

   Useful for: Quantifying the hypothesis validation, removing the sample bias

There are other popular methods that aren’t relevant or useful for your product most of the time like

— Eyetracking to see how users scan a product/page. It is good to understand how people read a page via eye-tracking. Here are some eye-tracking patterns. Eyetracking requires a sophisticated setup and quite time-consuming. It isn’t a method that produces a lot of insights apart from eye behavior. Here is how a sample eye-tracking heatmap looks like

— Focus groups are bringing 8-12 people in a room and getting them to discuss various aspects of the research. Focus groups are prone to groupthink and that’s why their insights are always questionable. Groupthink is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome.

— Diary study is used to understand the long-term behavior of the user. Users are asked to keep a diary and note their thoughts/usage patterns around a product. The data is self-reported. Diary studies have a high churn as a lot of people forget to do it over time. You need to offer incentives for diary studies, but as we discussed incentives can affect the outcome in unexpected ways. If done well, diary studies can help you how users ’ behavior attitudes change over time for your product.

— Card sorting is a method used to help evaluate how information should be presented on a site. In a card sorting session, participants organize topics into categories that make sense to them. You can use actual cards, pieces of paper, or sticky notes to do card sorting.

If you want to do one of these methods mentioned above, you can hire an agency to do so, or let UX research teams do it. If you are interested in reading about more methods quickly, you can read this article on UX research methods by Neilson Norman Group.

What you should focus on doing heavily are — usability testing, user interviews, and surveys.

Where do you start?

The quickest way to conduct user research is

— If you stay near your users and have access to them, you can go where your users are, talk and observe. I usually prefer to go to cafes and show the new design to a few people there and talk to them after buying coffee for them. This is provided the customers for the product also come to these cafes. If you are building a product for housewives, this method may not work.

— If accessibility is the problem, you can hire an agency to recruit these users and conduct the research with researchers/designers/business teams. A simple google search along with location like “user research agencies in Bangalore” will give you few good names.

If you want to do the user research quickly and know few users, there is one thing you must absolutely master before starting research — asking the right questions.

Asking the right questions

Let’s say you want to understand “why do people shop online?”

1. Ask questions which is easy to answer — If you ask users why they shop online, it’s a hard question to answer. So you have broken this question into smaller questions that are easy to answer.

2. Don’t ask leading questions”, i.e. questions that influence answer — When you try to rush to answers from users, you end up asking leading questions. A leading question is “How anxious do you feel while shopping online?”

3. Ask about specific instances — Specific answers are usually more accurate and insightful than generic ones, and one of the ways you can get specific answers is by asking about past instances when something happened. An example is “Tell me how you felt the last time you tried to buy something online and the purchase failed.”

4. Ask open-ended questions — Open-ended questions have a detailed answer. These cannot be answered with a simple 'yes' or 'no', and require the respondent to be detailed.

Putting an example below for you to understand and remember

Say you want to understand “why do people shop online?”

Ask questions which is easy to answer

“How do you buy different products?”

“What kind of products do you buy online?” —> “Why?”

“What kind of products you don’t buy online?” —> “Why?”

“What do you like the most about online shopping?”

“What do you hate about online shopping?”

Don’t ask leading questions

Change “How anxious do you feel when shopping online?”

to “Describe your online shopping experience last time”

Ask about specific instances

“Tell me how you felt the last time you tried to buy something online and the purchase failed.”

Ask open questions

Instead of “do people around you buy online?”

ask them “how do people around you purchase products?”

Final Words

I would leave you with 2 criteria that will help in deciding what important things have come in these user research sessions. A lot of time user research sessions lead to a lot of analysis-paralysis because you are trying to read too much into what every user. The actual meaningful insights are loud enough for everyone to hear and agree upon and can be triangulated. Let me explain

1. Loud insights - Building products on loud behaviors/insights from the research is always advisable. Loud insights are obvious to everyone.

   If there are some insights where everyone is interpreting them differently, you should either do more research to clarify or forget about them for the moment and act on the loud insights for the time being.

   If there are no loud insights, the research was conducted poorly and you should redo it. In absence of loud insights and triangulation, we end up doing this :)

   

2. Triangulation — Triangulation means using more than one method to collect data on the same topic. User research is just one source of data. Use product metrics, user psychology, and business understanding to back the user research. Suppose users are facing usability issues with a particular sign-up form, looking at the drops that happen on the form help you substantiate the usability testing.

Few words around consistency — It’s better to talk to 5 users every month than talking to 60 (5x12) in a month and then forget for another 12 months. There are two reasons why it is better

1. The first one is that consumer behavior changes rapidly. By talking to users every month, you are able to observe these changes.

2. The second reason is that your own understanding of the product and questions around change over time. So the research methods get more effective over time.

I will share some good resources next week that can help you to do a detailed study around interviews, usability testing, and surveys.

With this post, I wrap up Research and Experimentation: How to generate ideas and improve the metrics. Read the other 4 posts around this if you haven’t in the archives.

Sincerely,

Deepak

Twitter | LinkedIn