A New Approach to Applied Experiments
In this article, Connor Joyce, author of “Bridging Intentions to Impact” and CEO of Desired Outcome Labs, shares insights on designing effective experiments to achieve real results, even with limited resources. Known for advocating an Impact Mindset, Connor has extensive experience from his roles at Microsoft, BetterUp, Twilio, and Deloitte. He frequently contributes to various publications and is a seasoned keynote speaker.
Identifying the purpose of a feature is crucial for a team to align on design goals and measurement of success. Validating the effectiveness of a feature through experimentation allows teams to replicate successful features and market their impact, or refine underperforming ones to better meet user needs and business objectives.
Many teams struggle with experiments due to misconceptions. One is the belief that all experiments must be as rigorous as Randomized Controlled Trials (RCTs), leading to the notion that simpler methods are inadequate. Another is the idea that thorough digital infrastructure is a prerequisite for valuable experiments. While a full experimentation structure is beneficial, it is not necessary from the start. Additionally, many teams lack knowledge on how to set up experiments effectively.
This article and my book “Bridging Intention to Impact” aim to demystify digital product experimentation by defining a new standard and breaking down its seven essential components. Not all components need to be fully met to conduct valuable experiments. This guide empowers teams to design and execute experiments based on their current resources, providing actionable insights to advocate for further investment in experimental infrastructure.
In applied settings, RCTs are often impractical due to their resource demands. Teams should focus on practical experiments that deliver valuable insights and are feasible within their constraints. The ideal experiment for most teams is fully digital, allowing rapid testing and iteration essential in fast-paced development cycles. For instance, Airbnb runs hundreds of experiments annually using fully digital tests, allowing them to continuously optimize their offerings.
A fully digital experiment includes key components such as seamless data capture, broad participant reach, robust data infrastructure for real-time analysis, and advanced data science capabilities. Collecting both behavioral and attitudinal data provides a comprehensive view of user experience, enhancing the experiment’s value.
While RCTs are ideal in theory, applied settings require a pragmatic approach. By focusing on fully digital experiments and incorporating components like advanced analytics and diverse participant recruitment, teams can make informed, evidence-based decisions and drive continuous improvement.
Seven Essential Components of an Experiment:
1. Digital or In-Person Execution: Choose based on the product’s nature and required data. Digital experiments are typically more feasible, but in-person studies can offer valuable insights for physical products.
2. Experimentation Platform: An effective platform is vital for deploying features and managing variations efficiently, allowing for quick adjustments and iterative testing.
3. Data Infrastructure: Robust data systems are essential for seamless data collection, storage, and analysis, enabling real-time insights.
4. Data Science Capability: Advanced data science skills enhance the quality of insights through sophisticated analysis, adding rigor to the results.
5. Real-Time or Retroactive Data Collection: Decide based on the experiment’s objectives and resources. Real-time collection captures current interactions, while retroactive studies leverage existing data.
6. Attitudinal Data Collection: Using surveys and feedback tools to gather qualitative data complements behavioral metrics, providing a nuanced understanding of user impact.
7. Participant Pool: A diverse and representative sample reduces bias and enhances reliability. Recruitment strategies vary from broad digital outreach to building dedicated user groups.
Understanding these components allows teams to design experiments suited to their circumstances, driving meaningful insights even with limited resources.
Experiment Types on the Spectrum:
– Fully Digital Experiments: Ideal for reliability and actionability but require significant resources.
– Fully Retro Experiments: Use existing behavioral data, balancing fidelity and feasibility.
– Moderated Experiments: Involve direct user interaction in live settings, providing valuable insights with minimal resources.
Teams should assess their data needs and resources to select the most appropriate experimental approach. Whether striving for high-fidelity digital experiments, leveraging existing data, or employing moderated experiments for direct feedback, the key is aligning the method with objectives and constraints. By considering the spectrum of experimental options, teams can gather valuable evidence for informed decision-making, ensuring user-focused product development.