Is 'experiment' a good metaphor for startups?

I am Lean Startup fanatic. From the first time I read the material and heard Eric Ries speak it resonated with the geek in me and gave me insights that I have been using to help large organizations innovate like startups. I have also found that my background as a researcher and academic makes it easier for me to apply some of the tools such as customer development, cohort analysis and developing minimum viable products.

But something about the language that we use within Lean Startup has been bothering the scientist in me. It has been bothering me for a long time but more so lately, and I just can’t hold it in anymore:

An 'experiment' is NOT a good metaphor for describing startups…  

This issue has been raised before, but I think it may have been ignored. Defining a startup as an experiment does not really work in scientific terms. Firstly, because within your startup you will run a series of studies, of which only a few will be real experiments. Calling a startup an experiment makes it sound like a single event, when it’s more like an on-going series of studies, even after you achieve product-market fit.  

Secondly, most of what we do within Lean Startup just doesn’t meet the strict scientific standards that make the experiment the Holy Grail of scientific research. The experimental method is the Holy Grail because it is used to study cause and effect. After a researcher runs an experiment they should be able to say with some confidence that changes in variable X have a causal effect on changes in variable Y. To be able to reach such conclusions an experiment needs some of the following characteristics:

1. The ability to deliberately manipulate one variable (the independent variable), while holding other variables constant.
2. Some ability to deal with potential confounding variables that may affect your results. One way this is done within social science is to randomly allocate people to experimental conditions. Another way is to match research participants with regards to the variables you are trying to control.
3. The use of a control condition in order to take baseline measures upon which we will judge our results.

These rules/standards do not make experiments infallible. However, when an experiment is done properly following these rules gives researchers more confidence about the nature of the causal relationships between two or more variables. It also means that our experiments are replicable, which may be more important in proper science than in startups.

The Scientific Method
Looking at the above description, the only tool currently utilised within the Lean Startup that might meet the standards to be called experimentation is A/B testing, and this is only if the tests are properly designed to meet the above criteria. Cohort analysis may also be used as another powerful experimentation tool, especially if it is combined with A/B testing.

I think that the problem may lie in a confusion that views the scientific method as synonymous with experimentation. This is not necessarily the case. Experiments are only a sub-category of the scientific method. Developing falsifiable hypotheses is also an important part of the scientific method, but how you test those hypotheses is not necessarily with experimentation.

In Running Lean, which is a great book that I use in my own work, Ash Maurya defines an experiment as one full circle through the build-measure-learn loop. This is scientifically not correct. I think a successful circle through the build-measure-learn loop is better conceptualised as validated learning.  This is because you can go through the build-measure-learn loop several times, all the way to successfully achieving product-market fit without ever running a single experiment. However, you will run successful studies and you will learn a tonne about your customers’ problems and their perceptions of your proposed solution.

Indeed, Ash Maurya advises startups to validate qualitatively and verify quantitatively. This is wonderful advice that provides a practical actionable path through which entrepreneurs can take the right action at the right time. At the beginning, you really want strong signals from early adopters, so qualitative methods are appropriate. However, qualitative data collection is not usually associated with the experimental method. This does not make qualitative research non-scientific.  It is one of the tools within the arsenal of the scientific method. The same is true of all other research methods available to startups (e.g. case studies, interviews, surveys, user-testing, and customer observation). All these are powerful scientific tools that can provide valid data for startups to make informed decisions. But most of these cannot be described as experiments.

What Do We Want to Become?
The reason this issue has been bothering me is that I am concerned that the Lean Startup could grow into a pseudo-managerial-science that appears to use the scientific method and yet the practitioners do not understand scientific methods fully. This can create problems in terms of decision making. Each one of the research methods I cite above has its own limitations. If startups founders are unaware of these limitations they can make wrong decisions based on the data they collect. This can lead startups to use what they think are scientific methods and still fail.

For example, if your sample size is too small, and you didn’t use the right sampling methods, then you could pick up what you think is a strong signal from customers but is actually random noise. With small samples, the chances of this happening are actually quite high. This is why Ash Maurya encourages people to verify quantitatively the signals they get from qualitative research. But if people think their qualitative study is an experiment, with all the power associated with experimentation, they could feel more confident about their business ideas than they should. This could be particularly problematic for entrepreneurs who are already struggling with their reality distortion field.

In a recent post, Salim Virani raised an interesting question about how much the Lean Startup movement is learning from scientists and applying this learning to our work. I think we have choice during these early years of our movement. Do we want to fully align ourselves with scientific methodology and apply these methods appropriately to building startups? Or do we want to build our own approach with its own nomenclature and use methods that we don’t fully understand?  Is the full understanding of the scientific method really that important for startups? This is a choice we should make and be explicit about.

One approach is to say the Lean Startup method borrows from science a few things we find useful but our goal is not really to become a true management science for startups. I may be biased, but I think such an approach is just not good enough and will over the long term weaken the impact the Lean Startup method can have on entrepreneurship. This would be a real shame. Because until now there has been little opportunity to develop excellent methods for building startups that are teachable, replicable and scalable. I strongly believe the Lean Startup is just that method.