On the common fallacy of hypothesis driven business.

TL;DR Look before you leap.

There's a lot wrong with the world of software engineering but then again, there's always been a lot wrong with it. Much of this can be traced back to the one size fits all mentality that has pervaded our industry - be agile, be lean, be six sigma, outsource! 

However, there is a universal one size fits all which actually works. It's called look before you leap or in other words observe the environment before you decide to take any action. In the case of OODA loops there's even two whole steps of orientate and decide before you get from observe to the action bit. Alas in many companies action seems to be the default. Our strategy is delivery! Delivering what exactly? Who cares, deliver!

Your organisation, your line of business and even a discrete system consists of many components. All of those components are evolving through supply and demand competition from the uncharted space of the uncertain, unknown, chaotic, emerging, changing and rare to become industrialised over time. The industrialised have polar opposite characteristics to the uncharted something we've known about since Salaman & Storey's Innovation Paradox of 2002. If you want to accelerate the speed at which you operate and create new things then you have to break down complex systems into stable components and treat those components appropriately. 

So, how do you manage this? Well, since most companies fail to observe the environment then they will resort to the only thing possible which is backward causality or meme copying - "Everyone else is doing this thing, so lets adopt DevOps, Agile, Lean, Digital, Cloud, APIs, Ecosystems, Open Source, Microservices" and on and on. Each of these approaches have certain benefits if used in the right context but in most cases, the context is missing. Furthermore in today's software world various claims are given to being more scientific, to being driven by hypothesis but many of these ideas are misguided without context.

To understand why, we need to explore the game of chess. A key part of the game of chess is understanding the board i.e. where the pieces are (position) and where they can move to (movement). You don't actually have to physically see the board if you're good enough. You can create a mental model of the board and play the game in your mind. But the board is there, it's an essential element of the game. Though each game may be different, you can learn from each game and use these lessons in future games. This is because you can understand the context at hand (the position of pieces and where they can move) and can learn consequences from the actions you take. You can apply such lessons to future contexts. This is in fact how we learn how to play chess and why practice is so important.

Now, imagine you have no concept of the board but instead all you see is a range of computer characters on the screen (see figure 1). Yes, you can play the game by pressing the characters but you have no understanding of position or movement. Yes, over time you can grab the sequences of thousands of games and look for secrets of success in all those presses e.g. press pawn, pawn, queen, queen tends to win. You will by nature tend to copy other successful players (who also have no context) and in a world dominated by such chess play then memes will prevail - expect books on the "power of the rook". Action (i.e. pressing the key) will dominate, there is little to observe other than the sequence of actions (previous presses) and all these players exist in a low level situational awareness environment.

Figure 1 - Chess with low levels of situational awareness.

If you ever come up against a player who can see the context (i.e. the board) then two things will happen. First, you will lose rapidly despite having access to thousands of games containing millions of data points from sequences of action. Secondly you'll be bewildered. You'll start to grab for the spurious. Naturally, you'll try and copy their moves (you'll lose), you look for all sorts of weird and wonderful connections such as the speed at which they pressed the button (you'll lose), whether they had a good lunch or not (you'll lose) and whether they're a happy person or not (you'll lose). It's like the early days of astronomy where without any understanding we collected all sorts of things such  as whether it was a windy day. Alas you will continue to be utterly outplayed because the opponent has much higher levels of situational awareness and hence they understand the context better than you. To make matters worse, with every game your opponent will actually discover new patterns, new ways of defeating you and they will get better with time. I've tried to show an example of low vs high situational awareness in figure 2.

Figure 2 - low vs high situational awareness.

The player who understands the board will be absorbed by first observing the environment, understanding it (i.e. orientate and decide) and then making a move (i.e. acting). Terms like position and movement will matter in their strategy. Their strategy (the why of action) will be based upon why here over there i.e. why this move over that move. 

Most businesses exist in the low level situational awareness environment described by figure 1. They have no context, they are rife with meme copying, magic sequences and are dominated by action. We already know that this has an impact not only from individual examples but by examination of a range of companies. It turns out that high levels of situational awareness appears to be correlated with positive market cap changes over a seven year period (see figure 3).

Figure 3 - Situational Awareness and Market Cap changes.

So what has this got to do with hypothesis driven business? Hypothesis without context is often akin to saying "If we press the pawn button will it give us success?"

The answer to that question is it might in that context (which you're unaware of) but as the game changes with every move then there is nothing of real value to learn. Without understanding context you cannot learn patterns of play to use from one game to another. To give an example of this, let us examine The Scenario as described in an earlier post. This scenario has all the information you require to create a map and to start learning from previous conflicts and repeating patterns. However, most companies have no idea how to map and hence have no mechanism of past learning through context. 

It is certainly possible without context to create multiple hypotheses for the scenario e.g. expand into Brazil or maybe attempt to differentiate with a new feature? These can be created and tested. Some may well show a short term benefit. However, if you take the same scenario and map it  - as done in the Analysis post - then a very different picture appears. Past and repeatable patterns such as co-evolution, ILC & punctuated equilibriums can be applied and it shows the company is in a pretty miserable state. Whilst a hypothesis around differentiate with a new feature might show some short term benefit and be claimed as successful, we already know it's doomed to fail. The hypothesis therefore appears to be right (short term) but before acting, from the context, we already know it's wrong and potentially fatal (long term). It's the equivalent of knowing that if you move the Queen you might capture a pawn (i.e. success from a hypothesis of pressing the queen button) but at the same time you expose the King to checkmate (from looking at the context, the board).

The mapping technique described is about the equivalent of a Babylonian clay tablet but it's still better than having no map as it provides some idea of context covering position (relative to a user need) and movement (i.e. evolution). There will be better mapping techniques created over time but at this moment, this is the best we have. Many of us over the last decade have developed a sophisticated enough mental model of the environment, principles and repeatable patterns that we can just apply to them to a scenario without mapping it first. In much the same way that if you get really good at playing chess, you don't even have to look at the board. However, most have no understanding of the board, of position, of movement, of context or the numerous repeatable patterns (a subset of these, 61 patterns are provided below in figure 4). 

Figure 4 - An example list of repeatable patterns / gameplays

Without understanding context then most have no mechanisms of anticipation, learning and cannot even use weak signals to refine this. In such cases, you can make an argument that hypothesis driven business is better than nothing at all but it's a very poor substitute for understanding the context. Even if your hypothesis appears to be right, it can be completely the wrong thing to do.

This is the fallacy of hypothesis driven business. Without a mechanism of understanding context then any hypothesis is unlikely to be repeatable as the context will likely change. Yes, you can try and claim it is more scientific (hey, we've pinched a word like hypothesis and we're using data!) but it's the equivalent of saying "If I have a good lunch every day for a month then the heavenly bodies will move!" ... I had a hypothesis, I've eaten well for a month, look those stars moved ... success! Oh dear, oh dear, oh dear. Fortunately, astronomers also built maps.

This doesn't mean there is no role for hypothesis, of course there is! For example it is extremely useful for exploring the uncharted spaces where you have to experiment or for the testing of repeatable patterns or even for refinements such as identifying user needs. But understand the context first, understand where you can attack and then develop your hypothesis. The context is your route to continued learning.

Observe (i.e. look at the context) then Orientate & Decide (i.e. apply thought to that context) then Act (i.e. do stuff in that context).