UX Research Analysis
It doesn't matter what method you use to conduct user research, analysis should be the next step. You can't just take raw data and apply it to your product. Use analysis for sorting out the wheat from the chaff — i.e., defining what data is relevant and what isn't.
Regardless of the data type you analyze — web analytics, interview transcripts, field notes, or heatmaps — your focus should be on your research objectives. You should keep in mind your product's target audience.
Keeping this information at hand will help you stay focused during the analysis stage and define what's actually important for your project.
Performing UX
One of the biggest mistakes UX practitioners may make is jumping to hasty conclusions based on raw, statistical data and acting without proper consideration. Data analysis allows seeing the bigger picture, making more meaningful decisions, and thus, saving money and time. Instead of fixating on random numbers or users' quotes, UX researchers should seek an answer to why users behave or think in a certain way.[1]
For example, if your
Thorough data analysis helps teams clearly understand users’ pain points and make more rational decisions.
The analysis of
One of the disadvantages of attitudinal data is that users don't always do what they say they're going to do. This means attitudinal data can be confusing and lead to wrong assumptions at times.
Behavioral data is considered the most valuable data gathered during
Unfortunately,
Ideally, you should start thinking about the analysis phase early in the process — before the
Imagine your research goal is discovering why your customers abandon a shopping cart. In this scenario, you’d likely have certain assumptions about the rationale behind this behavior and would look for certain variables when analyzing research data.
At this stage, you can also create tags or codes that will help you categorize and identify data according to your research goals at the final stages. For example, if you're testing the usability of a landing page, you may create codes like "Navigation," "Aesthetics," "Critical errors," "Recommendations," and other pain points that you might discover during the study.
Analysis is necessary even in the discovery phase of your
Write down the words participants choose, their facial expressions, body language, and overall behavior — everything is important to get a deep understanding of their rationales, feelings, and needs.
Analyzing data in the discovery phase can save you a ton of time and resources before you work on the final analysis. In the long run, it helps you identify users' pain points and needs and create better products.[3]
Pro Tip! Make sure you have a 15-minute break between sessions to review your notes, discuss with your team, and write a summary.
Analyzing large amounts of data collected during the user study can be intimidating, and you might not know where to begin. Setting priorities and understanding your objectives makes it easier to focus on relevant data when analyzing results.[4]
Before you start, write down why you've decided to carry out
For example, you may discover your users are interested in having meditation practices in addition to yoga. These findings are "nice-to-haves" and can be put aside in the non-urgent category. On the other hand, "must-haves" should have the highest priority when analyzing. For example, if users are unsatisfied with the quality of your yoga videos, it must be fixed first.
Pro Tip! Keep the "nice-to-have" findings in your backlog so you can get back to them when you have the time and resources.
Once you've collected numerical data via surveys, polls, or web analytics, you will have a large dataset that can be analyzed with special software (SPSS, JMP, Stata, R, etc.) or simply with a spreadsheet.
Various methods can be applied to gain helpful insights from quantitative data:
- Cross tabulation allows
UX practitioners to analyze the relationship between multiple variables and determine patterns, trends, and probabilities within data sets. - Max-diff analysis allows measuring the preference/importance score of different items. It's often called the best-worst method as it helps identify what features matter most to your users.
- Conjoint analysis helps identify the optimal combination of features in a product or service, ranking them on a scale from most to least desirable. It consists of factors (e.g., class of service or airline brand) and its levels (economy and business; Southwest Airlines, Delta Air Lines, American Airlines, and United Airlines).
- Gap analysis demonstrates the difference between the desired state and the current state. For example, you can compare the actual and expected state of customer satisfaction.
- Trend analysis shows the change of a value over time and what parameters influence this change.[5]
- Sentiment text analysis helps researchers process users’ feedback from textual data using NLP (Natural Language Processing) tools.
The analysis of quantitative data implies working with numbers. Although it might appear a tedious and overwhelming task, quantitative
When analyzing quantitative data, you can define things like:
- The success rate of a specific task
- Time users spend to complete a task
- The bounce rate of a webpage
- Users' demographic profile
- Features that users use the most
- User satisfaction with a feature or product
- User needs that are not met by the product
- Critical features that require the greatest attention
- Different experiences of different user groups[6]
Qualitative research data deals with human behavior, which might be harder to analyze than numerical quantitative research data. It may take a lot of time to read long transcripts and extensive field notes and decide what details matter and what can be skipped. Also, participants' feedback can be conflicting, and researchers should remain objective and try not to ignore viewpoints that don't fit their beliefs. One of the best methods of breaking down and organizing rich data is thematic analysis.
Thematic analysis groups the collected data into themes by tagging individual observations and quotations with appropriate codes (like labels or keywords). While coding, researchers review each text segment and give it a name that describes the data. As they look for themes, some codes can be collapsed or expanded.
The last step involves evaluating your themes — a belief, practice, need, or another phenomenon that appears multiple times across data findings and can be supported with multiple instances.[7]
Thematic analysis can be performed using software, journaling (which involves manual coding and essential thought processes by researchers), or affinity diagramming techniques.
When analyzing data, keep in mind the following questions:
- What do users like most about this product?
- What do they like least about this product? Why?
- Which functions are more valuable?
- Which functions get unnoticed?
- Do they have an emotional response to certain features? When?
- Are they satisfied with the product? Why?
- How does the product fit into their daily lives? How important is this product to them? Why?
- What are the major patterns or common responses noticed in users' behavior?
Even the most experienced
- Having too much uncategorized data to stay objective. Ask your teammates for their opinions, prioritize user feedback, and decide what details can be omitted and what data is essential to be kept.
- Having biases about
research outcomes that can lead to misinterpretation of important data. To avoid biases affecting your results, write down all potential ones before the discovery phase and discuss them with your team. - Over-reduction, or "flattening," of the data gathered into close-ended survey responses like "yes or no" questions. Your focus should be on analyzing only the most meaningful and usable data. Also, make sure you include open-ended questions during
user studies .[8]
Once you're done analyzing qualitative and/or quantitative data, it's time to synthesize your findings and come up with insights. Synthesis helps researchers turn the identified themes into something meaningful.
Don't confuse findings with insights. A finding is a fact or a statement drawn from
An insight is a conclusion about
How do you synthesize findings?
- Prioritize findings and select the ones that seem the most relevant based on your
research objectives. - Organize your themes, findings, and insights using sticky notes, whiteboards, or any other system you prefer.
- List the most important insights in a document.
- Share them with your team and brainstorm together what can be learned from these insights.
Sometimes, different research methods can produce inconsistent or contradictory results. For example, the task success rate among users could be 100%, but they could still report that they're not satisfied with the app and would switch to another if they found a better alternative.
What can be done to prevent such conflicting findings? Start with checking the methodology:
- Respondents: Have the same respondents participated in both studies? Different people can respond differently, which can lead to contradicting results.
- Tasks: Have participants had the same amount of time for a task? Was the task the same for all participants?
- Environment: Have users been performing a task in the same environment, or was there something that could have distracted them or influenced their responses?
- Data analysis: Is the statistical significance high enough? Is there a chance that data was overcorrected?
If you find that none of your research methods have inaccuracies, you might have to consider conducting another study. Contradictory results are common, so don’t be discouraged.
The final step of analysis implies providing actual recommendations based on the key insights and supporting data. Including recommendations in your
You may also share a document with the list of gained insights and have an open team discussion in person or remotely. Instead of providing off-the-shelf solutions, you can
For example, the insight that "users abandon their
Insights and recommendations help your team define what to focus on and can be potentially turned into design solutions.
References
- How to Analyze Qualitative Data from UX Research: Thematic Analysis | Nielsen Norman Group