Sunday, August 09, 2015

Is there trouble brewing in the land of DevOps?

Back in 2008, I had a pretty well developed concept of how organisations evolved due to the commoditisation of component activities.  The process involves the co-evolution of practice and the interaction with cycles of economic change & inertia but I won't bore you with the details of how this works. Instead, I'll focus on an experiment we ran in late 2010 / 2011.

At that time, I had an opportunity to test whether these concepts of organisational evolution actually occurred or were just flights of past fantasy. There was the right sort of change occurring (due to cloud), the concepts anticipated a new phenotype of organisation would be emerging and we could use population genetic techniques to confirm whether this was happening. We struck gold. Across a reasonable sample size we showed that a distinct population (a next generation) was budding off from the main group. These two phenotypes are compared in figure 1.

Figure 1 - Traditional vs Next Generation Company.

This was all very exciting back in 2011 - well, in my specific niche field. It was the first time that I'm aware of that the process of organisational evolution had been anticipated and shown to be happening. I published the work in an LEF paper in 2011. However, this is all pretty standard fare today. For the purpose of this post, I'm going to focus on a few of those phenotypical changes - architectural practice, technique and structure.

In the past we had developed architectural practices around compute as product, things like N+1. As compute evolved we had developed new architectural practices around compute as a commodity (+utility). Hence the next generation had practices like design for failure, chaos engine, distributed systems (scale-out), continuous deployment and commodity infrastructure. This wasn't new in 2011, we had already given it a meme - DevOps.

The next change is the movement towards service / cell based structures which is related to the growth of structural approaches such as micro services, two pizza structures etc.

Finally, when it came to techniques then in the past companies had tended to one size fits all mentality. We were now seeing the emergence of mixed techniques i.e. the use of agile, lean and six sigma where appropriate.

Over the last 7 years the industry has continued to evolve pretty much exactly as expected with a continued movement towards utility services and a next generation of organisation. By 2011, we even knew the phenotype of that next generation with quite precise detail. All looks goods. Except ... trouble seems to be brewing down at the mill. Something odd has happened.

The first problem is related to containers. I'm not saying containers are a problem - except when it comes to sprawl and app containers - they are in fact an excellent future invisible subsystem focused on issues such as portability. The odd thing that has happened is related to configuration and package management. Somehow, and this is a more recent phenomenon, the idea that you don't need to worry about package management has appeared in certain quarters. Package management is just as important in a world of compute as a utility as it was in a world of compute as a product. Ignoring it has led to an issue that some IT landscapes contain components that people don't know how to recreate especially since the person that created the component has left the company. This is not healthy.

This problem is compounded when we discuss microservices. Again I'm not saying microservices per se are a bad idea, quite the opposite. The breaking down of large systems into components has been shown to provide significant benefits (e.g. the USAF FIST program). However, if we consider multiple layers of microservices then it can be quite disconcerting that some of those lower order component layers maybe in a position that we cannot recreate them. However, one advantage of a microservices approach is that our exposure is not significant and the component can be replaced. The clue is in the name - micro.

But here we come to a third issue, that of technical debt which is related to the change of organisations to use multiple techniques. Everything we do will evolve due to supply and demand competition i.e. every novel act becomes industrialised and more commodity like over time. Because of this evolution, the characteristics of the act, the techniques which need to be deployed (e.g. agile, lean or six sigma), the focus (exploring the unknown, reducing waste or empires of scale) need to change. If we don't deal with evolution we will build up technical debt which will ultimately slow our ability to make changes and build future higher order systems.  There are ways of dealing with this technical debt such as the pioneer, settler and town planner structure.

Unfortunately, what is happening with some microservices today is the debt isn't being dealt with and hence it's growing. We have environments containing layers of microservices where the underlying component services have not been industrialised but instead remain in a rather custom built state. Sometimes in more extreme cases, there is little or no documentation and the people who built the component have moved on.  In the worst cases, we're talking container like environments with no package management. This can become quite dysfunctional with developers building new microservices on top of nested layers of older microservices which no-one can effectively manage, no-one is industrialising and the debt is building. These companies are building the software equivalent of a house of cards and a few have started to see the consequences of this. However, coping with evolution, technical debt, package and configuration management are not new issues. 

Somehow, with all the good that is being done with DevOps, the desire to create the new world seems to mean that some are also ignoring those few older techniques that we know will be needed in this new world. It's like Columbus appearing in America and going "Ah ha, a new world, lets throw everything we know away and start again".

I'm afraid for some it seems there is trouble brewing in the land of DevOps.


Computer Repairs Sunshine Coast said...

It was such a good post. Visit computer repairs sunshine . Thanks for sharing.

Practice Management Software Australia said...

I agree with a lot of the points you made in this article. If you are looking for the Practice Management Software, then visit Practice Management Software Australia. I appreciate the work you have put into this and hope you continue writing on this subject.

shreekavi said...

This post is so interactive and informative.keep update more information...
German Classes in Chennai
German Classes in Bangalore
German Classes in Coimbatore
German Classes in Madurai
German Language Course in Hyderabad
German language classes in bangalore
German language course in bangalore
German courses in bangalore
Selenium Training in Bangalore
Software Testing Course in Bangalore

Ecare Technologies said...

Thanks for sharing this informative post with us...
Best Software Testing Institute in Bangalore

Gerry said...

Great post! We sometimes have to leave some old techniques in going to the future. site

Horea said...

Truly good! I am happy to see this here. Thanks!
click here

JAson96 said...

oes your organization use Agile project management? Are you deploying new capabilities on very short intervals to your infrastructure?

JAson96 said...

Nothing new here really. Shitty architecture and practices are shitty architecture and practices. If you're not disciplined enough about how you package, deploy, and manage the lifecycle of your software then putting the blame on technology X or methodology Y will not change anything.

Rajani said...

Thank you so much. It is a great article for me.
Django Online Courses
Django Training in Hyderabad
Python Django Online Training

Rajani said...

Hey Nice Blog!! Thanks For Sharing!!!Wonderful blog & good post.Its really helpful for me..
DevOps Training
DevOps Online Training
DevOps Training in Ameerpet

Mr Rahman said...

Really Nice Post & Thanks for sharing.
Oflox Is The Best Website Design Company In Dehradun

meldaresearch said...

Finding the best Help with Medical Assignment is not easy unless one is keen to establish a professional medical assignment help & medical homework help online.

Balajee Nanduri said...

Well Explained. Keep updating more and more DevOps Online Training

meldaresearch said...

Online medical research paper writing services are very difficult to complete and many students are always searching for Medical Assignment Writing Services to help them complete their medical coursework writing services and medical research paper services.

shane lee said...

With about 3 out of every 10 people being obese, more and more people are looking for information on safe diet pills that work. Resurge reviews

meldaresearch said...

Finding the best healthcare assignment writing services and Medical Assignment Writing Services is not easy unless one is keen to establish a reliable custom medical research paper provider & medical writing service help online.

lavanya said...

Thank for this blog. I got some knowledge in your blog it’s very useful for me now I started learning more testing on some other sites like. Software Testing Training in Chennai | Software Testing Training in Anna Nagar | Software Testing Training in OMR | Software Testing Training in Porur | Software Testing Training in Tambaram | Software Testing Training in Velachery

Anu said...

You have a good point here!I totally agree with what you have said!!Thanks for sharing your views...hope more people will read this article!!!
DevOps Training in Chennai | DevOps Training in anna nagar | DevOps Training in omr | DevOps Training in porur | DevOps Training in tambaram | DevOps Training in velachery

un known said...

Thanks for sharing this information. I really Like Very Much.
best devops online training

JAson96 said...

Does your organization use Agile project management? Are you deploying new capabilities on very short intervals to your infrastructure?

nisharoshan said...
This comment has been removed by the author.
nisharoshan said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..

Web Designing Course Training in Chennai | Certification | Online Course Training | Web Designing Course Training in Bangalore | Certification | Online Course Training | Web Designing Course Training in Hyderabad | Certification | Online Course Training | Web Designing Course Training in Coimbatore | Certification | Online Course Training | Web Designing Course Training in Online | Certification | Online Course Training