The experimental edge: How successful companies sandbox, pilot and test

By Simon Mueller, co-author of The Decision Maker’s Playbook (Financial Times Press, 2019)

Constantin is the third-generation owner and director of a niche marketing and market research company for the beverage industry. Clients hire him to assess and develop branding strategies for bottled and poured drinks. His mission: creating the desire that makes thirsty customers choose his clients’ bottles over others.

How do successful companies sandbox, pilot and test?

But what’s the best way to do so? What’s the right story to tell? What’s the most convincing concept? The answers to these questions have changed over time.

Designing bottles by relying on intuition and gut feeling used to be sufficient to succeed in the past. The market wasn’t crowded, and every professionally branded beverage was likely to capture a share of the pie. The market has become much more competitive since, and Constantin relies on data to make decisions: “Today, we have the tools to experiment and optimise a product down to the last detail. There are so many factors that could be the difference between buy and not buy: design, copywriting, colour, material. I don’t try to guess what could work. Instead, I’m simply putting a few thousand bottles out in various retail locations and collecting data to see what resonates most.”

Constantin runs business experiments. First, he creates a few variants of the final design, such as five different labels. Then, he asks several supermarkets to place his bottle on the shelf (he tries to ensure that the types of customers frequenting the supermarkets are some- what comparable). Finally, he uses the data he receives back to determine which design works best. This insight allows Constantin to start full-scale production of the design that has already proven to be highly effective in his experiment. Experimentation and testing can be an incredibly valuable tool in the arsenal of any decision maker.

Decision situations that are of substantial impact, but to a certain degree reversible, are the perfect application area for experiments. This is particularly true if the environment you find yourself in is complex or changing. The basic idea behind experimentation, trial and error is only feasible if the costs of error aren’t prohibitively high. If your potential choice is both important and hard to undo, take time to decide (for example, by applying scoring methods).

How to run business experiments

1) Define your hypothesis

What is it you are trying to find out? It is important to formulate your specific hypothesis before the start of the study. For example, your hypothesis might be that the modest increase in price (+10%) will not have a significant negative effect on your sales.

2) Define your outcome variable

Experiments allow you to establish a link between the change of an input factor (such as the price) on an outcome metric (such as sales). In plain English: Experiments help us understand what causal effect our actions have. Typical outcome variables are unit sales or revenues, click-through rates (if you are optimising websites), number of people cared for (in terms of housing, healthcare), self-reported quality of life, customer satisfaction and so on.

3) Create two groups: “Treatment” and “control” group

To measure what effect a ‘treatment’ (or intervention) has, you need two groups that share similar traits: one treatment group (receiving the intervention or treatment) and one control group (receiving no intervention, or only a placebo). It is important that both groups are selected randomly. Ideally, membership of any subject is determined completely by chance.

This is easier said than done. In many cases self-selection takes place, in that individuals sharing certain traits are more likely to be part of either the treatment or the control group. This skews the results, because certain shared peculiarities of the subject groups (such as socio-cultural factors, place of residence, or simply availability at time of experiment) can be the sole explainers for the difference between groups, rather than the treatment.

In order to achieve statistically significant results, you need a sample size that is large enough. There are complicated formulas to calculate the optimal sample size, but as a rule of thumb, 30 to 50 observations are typically deemed sufficient in business contexts.

4) Administer the ‘intervention’ to one group (treatment group)

After randomly assigning your subjects to either treatment or control group, it is time to adjust the input of the treatment group, while leaving the input of the control group unchanged. In our example, you would increase the prices of a given product (or product category) by 10% in 20 random stores around the country. Make sure you select a timeframe that is suitable to measure effects. A day would be too short, a year perhaps too long.

5) Measure and compare

After the pre-defined time period has passed, gather the output data (in our example, sales) and calculate the averages. If the assignment of samples to treatment and control group was indeed fully random, then the treatment effect (for a 10% price increase) is simply the difference between the average outcomes of the treatment group and the average outcomes of the control group.

When it comes to understanding causal relationships in business, we typically revert to speculation, mimicking others or best practices. But the results often disappoint, because best practices may work in one situation but not in others. Experiments allow you to test your solutions and find out if they survive the crash with reality.

The Decision Maker’s Playbook by Simon Mueller and Julia Dhar is out now, by FT Publishing, priced $21.99 (US) or £16.99 (UK). For more information go to: www.decisionmakersplaybook.com and see it reviewed here.

More information about Irish Tech News and the Business Showcase

FYI the ROI for you is => Irish Tech News now gets over 1.5 million monthly views, and up to 900k monthly unique visitors, from over 160 countries. We have over 860,000 relevant followers on Twitter on our various accounts & were recently described as Ireland’s leading online tech news site and Ireland’s answer to TechCrunch, so we can offer you a good audience!

Since introducing desktop notifications a short time ago, which notify readers directly in their browser of new articles being published, over 16000 people have now signed up to receive them ensuring they are instantly kept up to date on all our latest content. Desktop notifications offer a unique method of serving content directly to verified readers and bypass the issue of content getting lost in people’s crowded news feeds.

Drop us a line if you want to be featured, guest post, suggest a possible interview, or just let us know what you would like to see more of in our future articles. We’re always open to new and interesting suggestions for informative and different articles. Contact us, by email, twitter or whatever social media works for you and hopefully we can share your story too and reach our global audience.

Irish Tech News

If you would like to have your company featured in the Irish Tech News Business Showcase, get in contact with us at [email protected] or on Twitter: @SimonCocking