Experimentation is a discipline that many have heard of and are doing, however few have made the leap from traditional testing on an ad-hoc basis to optimise their digital properties to experimenting at scale across multiple facets of their business. This article discusses the attributes and factors synonymous with organisations that are known to have a healthy and successful culture of experimentation.

There is no doubt that the digital age that we live in has brought about disruption like never before where unknown small start-ups overtake established companies that have been in situ for decades, sometimes longer. In so doing, these start-ups have in some instances completely reshaped the way in which consumers buy and/or consume products and services, think Uber – when was the last time you called to arrange a taxi? But how do companies do it?

When looking at the attributes associated with these organisations’ success, the evidence is overwhelmingly clear that one thing alone stands out as being key to their success and that is the way in which they make business decisions. These organisations let their customers decide what’s best through the process of experimentation. Booking.com are widely recognised for their approach which sees them running thousands of experiments each year to inform and guide product related enhancements that move the needle on the dial that matters most to their business. Expedia are also a great example, they in their lifetime have conducted millions of experiments focusing on small iterative changes to their products.

These organisations have worked hard to determine what their success metrics are relative to their business objectives and have set about experimenting their way to achieving these. Their mindset and the associated culture that they have developed embraces the concept of experimentation and in so doing encourages employees to share new ideas, test them and roll out winning variations before further iteration. Netflix have compiled a set of guiding principles that makes this possible and have published these. They are the values that they as a business live by, which has enabled them to foster a culture that has led to Netflix being the disruptor that they are today.

There is no question that every business leader should aspire to develop such a culture within their own organisations, one whereby employees are empowered to not only do their jobs but do so in a manner which enables them to express their creativity.

One advantage that today’s disruptors have is their youthfulness, not in terms of employees or their age but they themselves are relatively young companies that had this culture within their DNA from the outset, something which more traditional organisations do not have, but instead have legacies and what one might consider corporate baggage which weighs them down.

Larger and more established organisations that have traditionally led the way within their respective markets may find it difficult to adopt this change and develop this approach since the fear of failure and employee job security are overwhelmingly dominant characteristics within such organisations therefore this change won’t happen suddenly. Employees need the right tools, training and reassurance that a healthy attitude to risk can and will be tolerated by senior management. This can only be achieved by way of a top down driven approach to establishing a collective vision that the company as a whole are prepared to buy into.

Define Your Change Agenda

Test, learn and fail fast should be a mantra that organisations employ and establish as a core value by which they live by. Maintaining a higher level of risk tolerance (calculated risk) is crucial in this instance and employees should be rewarded for their willingness to try as opposed to not try for fear of failure. As counter intuitive as it may seem. Failure should be recognised, promoted and evangelised about – since not every experiment is going to yield the results that might be expected, which is why we experiment in the first place – think how much time and money could have been wasted on an idea which wasn’t successful! Spotify’s Failure Wall is a great example of how to quash fears around failure which is described within this two part video

Whatever your new vision is as a business, it’s important to ensure that there is clarity and clout behind it i.e. that everyone understands what this vision is and what the associated expectations are, that senior executives within the business lead by example and are willing to not only evangelise about their own achievements but also what they have learned through experimenting.

Bind People with Technology using Robust Processes

Whilst it’s important to ensure that employees aren’t bogged down by bloated and bureaucratic process, some level of governance is crucial to ensuring that your experiments aren’t counter productive or conflict with other experiments conducted by adjacent departments.

Ensure that your business has the right people that possess the necessary skills. Data and numbers are important therefore being able to interpret results and moreover draw a conclusion that is both honest and one that tells a story is very important. Furthermore, ensure that everyone understands your business’ manifesto around experimentation to ensure that employees are motivated and are excited about what your business is trying to achieve with this step change.


Businesses world over are blighted by acronyms and terminology that only they themselves understand and it gets worse, such confusion is also present at a departmental level. The premise of experimentation is to get multi-disciplined and traditionally siloed teams working together to solve problems and drive innovation and therefore its important to ensure that everyone understands one another. A company wide dictionary filled with business specific definitions should exist which not only informs and educates but further defines what is meant by each term relative to your business. For example, Your KPIs should be understood by everyone, the relationship that exists between them should also be understood and acceptance criteria for matters such as statistical significance should also be agreed upon upfront, to avoid conflict when it comes to analysing results and defuse tensions when experiments don’t yield their desired results or go to plan. Arguing over which statistical model to use once an experiment has been completed is counter-productive in that more effort is wasted on measuring than on the object of the experiment. Chose a model and stick with it – irrespective of whether anyone’s ideas are “winners” or not.


Lastly, equipping your employees through the democratisation of data and tools is paramount to success. To achieve true scale employees need to be empowered and feel confident about interpreting data to tell a story. Dashboards and reports should be created and usage should be monitored to ensure that those who ask for the data do in actual fact use it. There is little value in your analysts creating and circulating reports their whole lives. Their time should be spent delving into the data to support your understanding of your experiments, why certain events occur and experiments perform they way they do, so that you can make informed business decisions. Enabling and encouraging staff to seek out the answers they need in order to be able to conduct their own experiments not only makes everyone more productive, but also drives the groundswell of momentum required to establish a culture of data driven experimentation. If time and training are key requirements, ensure that these are provided.

Equipping isn’t just about tools and training. In order to foster a culture of experimentation that leads to innovation employees at all levels need to feel safe and rewarded for looking at new ways of solving problems and/or further enhancing your products. Consider an internal innovation programme that promotes this behaviour amongst. A great example of an organisation that has taken this notion a step further is Adobe, who have an internal programme running that aims to drive innovation called Adobe Kickbox which has led to unlocking significant potential within their business.