At Starttech Ventures, we are big proponents of the Lean Startup methodology. One of its cornerstones is validated learning—a concept that emphasizes learning through real-world data and experiments to inform better decisions. For us, marketing experimentation is one of the most powerful tools to achieve this learning. Whether it’s through A/B testing, multivariate testing, or campaign optimizations, each experiment provides us with valuable insights that help refine our marketing strategies and grow our businesses.
Yet, it’s surprising how often, in meetings or discussions, I hear experiments being categorized as either a “success” or a “failure.” From my first-hand experience, I know this binary classification can be harmful and misleading. Marketing experiments aren’t about success or failure—they are about learning. Let’s explore why viewing experiments as learning opportunities, rather than win-or-lose scenarios, is essential for marketing success.
Breaking Free from the Success/Failure Trap
All too often, marketing experiments are framed in terms of “winning” and “losing.” If a test version outperforms the control, it’s labeled a success. If not, it’s deemed a failure. This mindset oversimplifies what experiments are actually meant to achieve.
Marketing experiments should never be about guessing the right outcome; they are about discovery. Even when a control version “wins” over the test version, it doesn’t mean the experiment failed. What truly matters is whether the data produced by the experiment is reliable and whether we can extract actionable insights. For instance, if you test two landing pages and find that a simpler design drives more conversions, you’ve gained a valuable insight, regardless of whether the result aligned with your expectations.Adopting a mindset of resilience in the face of unexpected results is essential. Like the “Good” approach championed by Jocko Willink, we can look at any surprising or disappointing outcome and think, “Good—there’s something here to learn.” A “failed” experiment often reveals areas where our understanding is lacking or where there are fresh insights we haven’t yet considered. With this approach, setbacks and unexpected results provide us with more opportunities to deepen our insights.
Success Defined by Learning: The Lean Startup Approach
The Lean Startup framework revolves around the concept of validated learning—where the ultimate goal of any experiment is to learn something valuable. This perspective shifts the focus from outcome to insight. If an experiment yields reliable data that teaches us something about our audience or our strategies, then it is successful, regardless of which variant performed better.
Let’s consider an example. You run an A/B test on an email campaign with the hypothesis that personalized subject lines will perform better than generic ones. The test results show no significant difference between the two. Is that a failure? Absolutely not. What you’ve learned—that personalization doesn’t strongly influence this particular audience—can save time and resources on future campaigns.
True failure in marketing experimentation only occurs when an experiment doesn’t provide reliable data, perhaps due to sample size issues, biases, or poor test design. As long as the data is actionable and helps inform future decisions, the experiment is a success. In marketing, learning is the real victory.
Language Matters: Moving Away from Binary Labels
The language we use to describe experiments has a powerful effect on how we approach them. When we talk about experiments in terms of “winning” and “failing,” we set up expectations that don’t align with the actual goal of experimentation. This miscommunication can lead to frustration, or worse, it can result in abandoning valuable insights because the test didn’t meet expected outcomes.
Instead, we should think of every experiment as an opportunity to learn. In my experience, this shift in mindset can dramatically improve how a marketing team approaches its strategy. Precision in language helps everyone—from marketers to stakeholders—understand that learning is the ultimate goal, not just proving one variant is better than the other.
Coming from a scientific background, I’ve learned to value the importance of language and clear communication. Even though I spent much of my education in the sciences (in Greece’s 1st desmi track), I’ve come to recognize the essential role that language plays in experiments. Precision in defining outcomes is just as critical in marketing as it is in the sciences.
Best Practices for Learning-Driven Marketing Experiments
To make the most of your marketing experiments and ensure they lead to valuable insights, follow these best practices:
- Start with Clear Hypotheses: Define what you want to learn from the experiment. This will keep the focus on learning rather than merely picking a winner.
- Prioritize Credible Data over Frequentist Significance: Instead of relying on traditional statistical significance thresholds, use a Bayesian approach to interpret results as probabilistic and adaptable to the context. Bayesian methods can help provide a clearer, more flexible perspective on the likelihood that your findings are real and relevant, enhancing confidence without rigid thresholds.
- Iterate Based on What You Learn: Use insights from each experiment to guide your next steps. Every experiment should lead to more informed decision-making.
- Document Everything: Maintain detailed records of your experiments, including outcomes and key takeaways. This documentation will help ensure that the insights you gain are shared and applied effectively.
In the world of marketing, it’s tempting to focus on results and categorize experiments as either a “win” or a “loss.” But when you approach marketing experiments with the mindset that they are opportunities to learn, you unlock the full potential of validated learning—a core principle of the Lean Startup approach.
As fellow Greek and NBA star Giannis Antetokounmpo once said, there is “no failure in sports”—only steps along the path to success. This perspective resonates deeply with us at Starttech Ventures. Each experiment, regardless of its outcome, is simply another step toward greater understanding and long-term achievement. The real measure of progress isn’t in whether we “win” each test but in the insights we gain, the reliable data we gather, and how those lessons shape our strategies going forward.
And if this mindset is championed by none other than the Greek Freak himself, then maybe it’s time we all embrace it.
