Analyzing qualitative feedback can feel like an immense task, with dozens of hours of video and hundreds of pages of transcripts from which to extract insights.
The good news is that you’re not doing rigorous academic research, so there are shortcuts that simplify this task significantly. Most market research reports basically consist of consolidated and organized feedback to a set of well defined questions as defined by the research brief, making analysis relatively straightforward.
A PROCESS FOR ANALYZING QUALITATIVE DATA
I’ve found the following process to work well for delivering insights from interviews in projects where (a) you start with clearly defined research questions, and (b) you have enough context to make informed calls on what the feedback means.
This is by no means the only way to do qualitative analysis, but it’s a process that works for most projects.
- Before you start your first interview, create the analysis scaffolding.
- Create a new document, called something like “Consolidated feedback”. Add each interview question as a separate heading, grouped by theme.
- Create a second doc, called “Analysis”. Add each research question as a heading (the internal questions that you want to understand with this research, not the individual interview questions).
- Immediately after each interview, copy/paste the response to each interview question from the transcript into the ‘Consolidated feedback document under the apprporiate question heading.
- Keep name and time stamp from the transcript, so you know where the comment originates.
- Add any new questions you asked in the interview.
- Once you have completed all interviews, and have all responses organized in the ‘Consolidated feedback’ document, that is the basis for your analysis.
- Read through the feedback, question by question, for commonalities and differences between responses. If you were to describe what you found to a colleague – what would you focus on, and how would you define what you heard?
- Pull out how different segments answer the questions. It’s valuable to note both where feedback is similar between segments and where it differs.
- Note where feedback within a segment is more complex or internally diverse than anticipated.
- Consider if the feedback supports or contradicts the hypotheses in the research brief.
- Identify quotes that speak to the themes you’ve identified.
- Add your observations into the Analysis document under the appropriate research question headings. Add in relevant quotes.
- When you have gone through all of the interview questions and written up your observations, that is the core of your report. At this point, you’ve pulled out 75% of the insights.
- Now it’s time to get at those last 25%.
- Read across your question-level observations in the Analysis document, and pull out themes that come from looking across multiple questions. This is often macro-level insights about segments – in addition to the question-level observations, what are the more holistic insights on segments differences or similarities that you can codify.
- In the ‘Consolidated feedback’ document, read all respondent answers in each section in one go. Note feedback that makes you go “huh, that’s interesting” even if it’s not directly relevant to a research question.
- Maybe participants end up comparing your product to competitors, even if you didn’t ask. You had not intended to collect competitive intelligence, but now you have a treasure trove of feedback. Use it.
- Leave the analysis alone for a few days, to give your brain time to make new connections. Return and do another scan of both documents, and add any new observations. It makes a big difference to have some distance from your first pass.
- Now you should have a comprehensive analysis, structured as observations to your research questions. The next step is to clean up the analysis document and make it into a research report.
Recommendations
- Quotes help a lot with color and stakeholder buy-in – a report with direct quotes will be much more engaging and memorable. Try to include quotes for any major points.
- Be aware of your own bias – examine your assumptions and make sure you put as much weight onto feedback that contradicts your assumptions as on the ones that support them.
- Don’t fall in love with outliers – often you get one respondent with a different view than the rest. It’s worth noting the disparity, but don’t center your analysis around outliers.
- But don’t ignore the outliers – inconsistency is inherent to any market. Don’t simplify your analysis so that you create a perception of coherence where there isn’t one. Report the 90/10 split, but make recommendations that focus on the 90%.
- Clean up quotes for legibility – verbal transcripts are messy. For internal reports, don’t hesitate to tweak grammar and sentence structure in quotes to make them read easier, as long as you don’t change the meaning.
- Video clips sell quotes – video quotes really help bring feedback to life. Be careful to not introduce bias, as any point which is supported by a video will be remembered more vividly than ones without a supporting clip.
- Don’t use research quotes for marketing purposes – your marketing team might want to use a quote of a customer lavishing praise on your product in marketing material. Don’t let them – your respondents expect their feedback to be for internal use. If you think a participant is amenable to being used in marketing, start a new conversation thread where you ask for an on-the-record quote.
A note on alternative qualitative analysis methods
I like the method outlined above because it’s fast, repeatable, and offers a framework for reliable analysis. But it’s not the academic way to do qualitative analysis, and it does rely on you having enough subject matter expertise to pull out relevant insights.
The common method for qualitative data analysis in academia and some UX research involves ‘coding’ the interview transcripts – assigning labels to each piece of feedback, and then organizing these codes into a conceptual framework. This is a useful method for when you have no a priori assumptions about the results. You can find articles describing this type of analysis here and here.
A challenge with this type of analysis is that it is very time consuming. If you need to crank out a report in a week, you can’t spend the first 200 hours coding up your transcripts.
WRITING THE REPORT
Expanding your analysis into a full report will make it a standalone read that anyone in the company can get value from. The report includes the context for why the research happened, how it was conducted, and what recommendations people should take away.
A standard report will contain the following:
- Executive summary – this will be the last part you write. It should be detailed enough to be actually useful, while still scannable in a few minutes. Think about what you would include if you had five minutes to explain the research.
- Background – this should give context for what’s to follow. It covers what led you to doing this research, past research, stakeholders, etc.
- Methodology – a brief description of how the research was conducted, time of data collection, etc. Include a link to the research brief.
- Respondents – a table of all interview participants, company, title, and key information or segments (e.g. ‘active users’, ‘churned user’), plus links to interview transcripts and videos.
- Analysis – add in your observations here.
- Hypotheses results – this section lays out all the assumptions in the ‘hypothesis’ section of the research brief, and maps your results against them.
- It’s useful for communication to color coding the results (e.g. green for “Hypothesis supported”, yellow for “Partially supported”, red for “Rejected”).
- Recommendations – a section with concrete, actionable ideas for changes coming out of this research. There is section below on how to develop recommendations.
- Next steps/further research – often a research project ends with open questions or with ideas for further investigations. Outline what you want to do and why, as a way to get buy-in for further research.
MAKING RECOMMENDATIONS
It can be challenging to draw insightful recommendations from your analysis. Often you might not know enough about the teams and products that your recommendations would impact, and it can feel presumptuous to tell stakeholders what your results mean for them. But the recommendations section is important – it is where research delivers value – so don’t skip it.
There isn’t a clear process for developing your recommendations section as there is for the analysis, but here is general advice:
- Consider your recommendations a starting point – your list of recommendations is a way to engage stakeholders to think about what actions to take from the analysis. You don’t need to outline every action yourself, but you provide the kindling.
- Feel free to add caveats – when sharing recommendations with stakeholders, you can add in a caveat that your view is based on limited insights into their plans and products. This allows them the freedom to reject unworkable ideas without it being a big deal, opening up the space for better positive-sum conversations.
- Review your analysis for business implications – a good way to find useful recommendations is by extrapolating from your analysis into what business changes are implied.
- Tie recommendations to business objectives – there was a reason you conducted this research. Map recommendations back to the business objectives in the research brief.
- Highlight opportunities – your ideal recommendations should be forward-looking and identify unexplored revenue or growth opportunities. Tie recommendations to business value.
- Review transcripts for respondents ideas – you can get many recommendations directly from your interview participants. Find themes where you’ve heard several respondents suggest the same thing, and rephrase in your own words.
- Use direct quotes if possible – if you can find a respondent quote that speaks to your recommendation, that is really powerful.
- Include obvious recommendations – some recommendations will be basic and it’s tempting to skip them. Include everything, as it’s useful to have a record of all ideas, and what seems obvious to you might not be such to others.
- Group the recommendations by team responsible for implementation – if you have multiple stakeholders from different teams, ensure they can easily find recommendations that are relevant to them.
- Be ambitious – include unlikely ideas that are large or long-term in scope. You can always caveat them, but it’s good to follow your analysis to its logical conclusion.
- Stack rank or prioritize your recommendations – if you have a long list of suggestions, make sure to rank them on e.g. feasibility, ease of implementation, impact, etc. Don’t just present an unfiltered raw list.
OTHER POSTS
The guide includes the following posts:
Photo by Jonas Jacobsson
