Back in 2008, I had a pretty well developed concept of how organisations evolved due to the commoditisation of component activities. The process involves the co-evolution of practice and the interaction with cycles of economic change & inertia but I won't bore you with the details of how this works. Instead, I'll focus on an experiment we ran in late 2010 / 2011.
At that time, I had an opportunity to test whether these concepts of organisational evolution actually occurred or were just flights of past fantasy. There was the right sort of change occurring (due to cloud), the concepts anticipated a new phenotype of organisation would be emerging and we could use population genetic techniques to confirm whether this was happening. We struck gold. Across a reasonable sample size we showed that a distinct population (a next generation) was budding off from the main group. These two phenotypes are compared in figure 1.
Figure 1 - Traditional vs Next Generation Company.
This was all very exciting back in 2011 - well, in my specific niche field. It was the first time that I'm aware of that the process of organisational evolution had been anticipated and shown to be happening. I published the work in an LEF paper in 2011. However, this is all pretty standard fare today. For the purpose of this post, I'm going to focus on a few of those phenotypical changes - architectural practice, technique and structure.
In the past we had developed architectural practices around compute as product, things like N+1. As compute evolved we had developed new architectural practices around compute as a commodity (+utility). Hence the next generation had practices like design for failure, chaos engine, distributed systems (scale-out), continuous deployment and commodity infrastructure. This wasn't new in 2011, we had already given it a meme - DevOps.
The next change is the movement towards service / cell based structures which is related to the growth of structural approaches such as micro services, two pizza structures etc.
Finally, when it came to techniques then in the past companies had tended to one size fits all mentality. We were now seeing the emergence of mixed techniques i.e. the use of agile, lean and six sigma where appropriate.
Over the last 7 years the industry has continued to evolve pretty much exactly as expected with a continued movement towards utility services and a next generation of organisation. By 2011, we even knew the phenotype of that next generation with quite precise detail. All looks goods. Except ... trouble seems to be brewing down at the mill. Something odd has happened.
The first problem is related to containers. I'm not saying containers are a problem - except when it comes to sprawl and app containers - they are in fact an excellent future invisible subsystem focused on issues such as portability. The odd thing that has happened is related to configuration and package management. Somehow, and this is a more recent phenomenon, the idea that you don't need to worry about package management has appeared in certain quarters. Package management is just as important in a world of compute as a utility as it was in a world of compute as a product. Ignoring it has led to an issue that some IT landscapes contain components that people don't know how to recreate especially since the person that created the component has left the company. This is not healthy.
This problem is compounded when we discuss microservices. Again I'm not saying microservices per se are a bad idea, quite the opposite. The breaking down of large systems into components has been shown to provide significant benefits (e.g. the USAF FIST program). However, if we consider multiple layers of microservices then it can be quite disconcerting that some of those lower order component layers maybe in a position that we cannot recreate them. However, one advantage of a microservices approach is that our exposure is not significant and the component can be replaced. The clue is in the name - micro.
But here we come to a third issue, that of technical debt which is related to the change of organisations to use multiple techniques. Everything we do will evolve due to supply and demand competition i.e. every novel act becomes industrialised and more commodity like over time. Because of this evolution, the characteristics of the act, the techniques which need to be deployed (e.g. agile, lean or six sigma), the focus (exploring the unknown, reducing waste or empires of scale) need to change. If we don't deal with evolution we will build up technical debt which will ultimately slow our ability to make changes and build future higher order systems. There are ways of dealing with this technical debt such as the pioneer, settler and town planner structure.
Unfortunately, what is happening with some microservices today is the debt isn't being dealt with and hence it's growing. We have environments containing layers of microservices where the underlying component services have not been industrialised but instead remain in a rather custom built state. Sometimes in more extreme cases, there is little or no documentation and the people who built the component have moved on. In the worst cases, we're talking container like environments with no package management. This can become quite dysfunctional with developers building new microservices on top of nested layers of older microservices which no-one can effectively manage, no-one is industrialising and the debt is building. These companies are building the software equivalent of a house of cards and a few have started to see the consequences of this. However, coping with evolution, technical debt, package and configuration management are not new issues.
Somehow, with all the good that is being done with DevOps, the desire to create the new world seems to mean that some are also ignoring those few older techniques that we know will be needed in this new world. It's like Columbus appearing in America and going "Ah ha, a new world, lets throw everything we know away and start again".
I'm afraid for some it seems there is trouble brewing in the land of DevOps.
