## Experiments #### Why Experiments? When building new features, it’s important to start with a **problem statement**. This is generally a simple guiding principle that makes it clear _what problem_ a particular feature is going to solve. We don’t want to invest time in features that are unnecessary. We want to invest our time wisely, and build features that add value by solving customers’ problems! More often than less, the right answer to this question is to try out several different things, see what proves most effective by following the data, and let our end-users tell us, implicitly, with their behaviours, what the best experience is. Simply put, we need to run an experiment or A/B test; we need to show end users many different experiences, and follow their behaviours to figure out which experience is the best. #### Vocabulary: All good experiments have a couple of important attributes: 1. **Control Group:** A cohort of users who are not part of the experiment at all. 2. **1 or more variants:** Different features being tested within the experiment. C: Apple Pay B: 1-click payment A: automatic billing 3. **Dependent Variables:** Metrics that are likely to be influenced by the experiment. They are the key things we are optimising for. It is used to prove hypothesis. - So let's say we are implementing a new checkout flow and have three different variants with group of people trying them, and we have a hypothesis that the "1 click payment" method will improve the overall check out experience, in that case the "time to convert" metric, within our **checkout funnel** is a dependent variable of the experiment. - So, dependent variables are the set of metrics that our experiment is intended to change. ![[PMing_Analytics_Funnels Segments.png]] 4. **Independent Variables:** The attributes which cannot be influenced by the experiment. - Attributes that have nothing to do with what we are testing. (eg. user's age, gender, location and other such data that does vary across our user population, but cannot be affected by our experiment) - Independent variables are generally used to ensure that the experiment variance are **well distributed** and there are no unintended factors which might confound the results of our experiment. - It helps avoid any form of biases in the experiment which might lead to wrong conclusions. - The goal of quantifying independent variables is to ensure that each group of users in our experiment is similar enough to our entire user population, such that any form of bias is not introduced within the data. ![[PMing_Analytics_Bar Chart.png]] #### Cycle of Analysis: 1. Frame a question 2. Identify dependent variables (turning it into a hypothesis) 3. Develop a success metric (translating variables into concrete metrics) 4. Break the results in the data out by the experiment group (to get conclusive results) --- `Conclusion: not only have you learned how to build and analyze your very own custom metrics, you’ve done a great job of navigating complex business requirements and working with different types of teams—all which have their own priorities.` Often, product analytics is about adapting your strategy by framing questions, making hypotheses, testing those hypotheses with proper analysis, and using the results to frame new questions. You can think of this entire process as a wheel, and as you turn the wheel, you get new insights to help you make better decisions.