In this post I’ll discuss an overview of how to best utilize data, analytics, and experiments such as A/B testing, when launching a new feature. This is what I often see neglected when designers and stakeholders are caught up in wireframes, animations, and coming up with the next big feature.
Using the Scientific Method
Lean startup, MVP, Design Thinking, UX Process-- at their core are arguably all rebranded ways of saying the scientific method, which often becomes the last step of the design process or gets completely neglected. The scientific method is a continuous process that involves the definition, and testing of a hypothesis through an experiment, resulting in measurable evidence. The measurable evidence is then compared to the predicted outcome, often times resulting in a new hypothesis, and experiment to test. So, how does this apply to designing a new feature?
Defining your Hypothesis
A feature should always be broken down into it’s smallest core part, as to make measurability more accountable. On one project I was tasked with designing the UX for adding a new comments section with the ability to Up-Vote comments, Reply to comments, and tag users in the comments section. The core hypothesis here, is that “Users will write comments”. If users don’t leave comments in the first place, there will be no comments to read, reply to, tag users in, or up-vote, so it is critical to optimize the commenting experience before adding on the rest of the features.
It’s often the case that breaking down a feature to this bare-bones level receives a lot of push-back from stakeholders such as the client, so it is important to weigh the trade-offs, as each feature and each situation is unique. For example, will adding the ‘up-vote’ feature hinder your ability to optimize the core hypothesis of leaving comments? Probably not, if that feature is designed as a secondary call-to-action, and doesn’t conflict with the core hypotheses visual hierarchy. Will adding the ability to reply to comments hinder this optimization? Same answer, but now we also need to consider that it will slow down the development and design time, time that could have been spent gathering user analytics data through an earlier launch of a smaller feature, data that could have been used for optimizing. This means that your engineering and design resources, and access to market are a factor too.
Measuring Evidence
Once the core hypothesis of the feature is defined, it should always be paired with at least one quantifiable way to measure it’s success, through an analytics tool that does event tracking-- my preferred tool is MixPanel. These Key Performance Metrics should be defined before wireframes are made, this will keep the designs oriented around the correct business goals, and make sure that the wireframes aren’t done for the sake of making wireframes. Here are examples of what should be measured:
-what % of users tap the comment button
-what % of users start entering a comment
-what % of users post a comment
-what % of users get into the ‘see all comments’ page
These data points, as with most user analytics data points, should be analyzed separately for different segments such as new users, and returning users, as they could have drastically different experiences. These results will reveal some gaping holes and opportunities in the feature’s design and optimization, which should ideally reach a comfortable level in a private beta setting before launching the feature on a mass level.
My favorite thing about this process is that you and your coworkers can be on the opposite sides of the hypothesis, and the data will determine a real winner. I have been in situations where the stakeholders did not respect the process and data, and tried to interpret it each in their own way, but that’s a bigger cultural issue.
In future posts I will cover how A/B tests and usability tests can be utilized as a part of this process.
