Thursday, December 03, 2015

Two speed IT - the more I look, the worse it gets.

I've just been told of a "new" idea of using two speed IT with agile combined with a platform. Heaven help us. First, it's a very old idea and second it was as daft then as it is today.

Let's go through how things should work (see figure below). Your town planners (specialists in empires of scale) create your platform. Lets start with a simple code platform providing a utility like service. Your pioneers (the specialists of speed) build lots of new stuff on it. Your settlers (the specialists of learning, reducing waste) find the common patterns in all the new stuff and turn these into basic components whether products or libraries which they develop further. Let us be clear here, whilst your settlers may take "more" time to build a product or library component than your "hack it anyway it works" specialists, your settlers accelerate the speed of everyone building with those components. Hey, I just install the library rather than write one or there's an API to some rental like service (which is fairly stable).

Over time, your town planners convert these increasingly matured components into component services optimised for volume operations. Let us be clear here, these component services become extremely stable, change slowly (though new component services can be constantly added) and provide volume operations of good enough. Whilst the services may reliably provide volume operations of good enough, what "is" provided can be a highly standardised, less performing and less reliable component than available from the most super duper products. The key is volume operations, good enough, optimised for delivery and a very low mean time to recovery (MTTR) i.e. if the component fails, another is available rapidly. This forces architectural practice change (see Devops) but also the standardisation, change of focus, speed of delivery helps accelerate the speed of everyone who builds upon them.  For your pioneers, they don't have to install a library or product but instead they just use an API to a solid service which is super fast ... but they have to design for failure! It also helps your settlers who can include these component services in their libraries and products.

And so the cycle goes on and on, constantly driving us upwards and ensuring efficiency, agility, speed and removal of debt by the use of three groups, three speeds of operation, three types of people and three cultures. This was reasonable practice circa 2005.

So lets roll the clock forward to 2015. What we have is a "great" idea of one group builds the platform and then a second "agile" group running at a different speed builds the new things (see figure below). Certainly this might give you a short term bump in efficiency and delivery. The problem is there's no group identifying common patterns and evolving them. The "platform" group that builds industrialised component services aren't going to touch your flaky new "agile" thing (cue endless arguments to exacerbate any tension between the two groups). The agile team of course wants to move on, so they'll start building new things on top of their new things. This is a one way street to spaghetti junction, a bucket load of technical debt, an entire system that creaks and slows downs and a big "meeting" where we all get together to build the new platform of the future because the last one is now fscked. I've seen this so many times over the last decade that it ain't funny.

Two speed IT combining agile + platform ... that's my new shibboleth for the clueless. Almost anything is better than a big expensive platform effort which will grind to a halt requiring a huge future platform re-engineering effort and on and on. This is where you'll be heading and this is kids stuff. It's shockingly bad.

-- 6th January 2016 : On Speed
[added updates above]

Lots of people discussing the importance of speed. Please note, don't confuse the speed of how quickly something is built with the overall impact of the thing on speed i.e. it might take longer to build a library than a quick hack for the same activity but the advantage of the library is in re-use. The more industrialised something becomes then the speed of building / changing might become slower but the speed benefit it has on everything that is built on top becomes huge.

It's way quicker for me to chop down a tree and saw a plank of wood than it is to build a modern computerised sawmill. But if I need planks for a 100,000 homes then it's way better to have highly industrialised and standardised planks created by computerised sawmills than to have me with an axe and a saw.

-- 5th April 2016 : On Legacy
[added updates above to make clear the point]

Some people seem to think that "legacy" belongs with the town planners. Legacy is often the result of best architectural practice for consumption of a product which is different from best architectural practice for consumption of a utility. The issue is that architectural practice tends to co-evolve with the activity, so when we shift from product to commodity (+utility) then we get novel, then emerging, then good and finally best architectural practice for a commodity (+utility). In IT, we call this DevOps and I've written more on this subject in an earlier post.

Of course, though we experience this change of practice, we should be mindful that we've previously been through a cycle of novel, emerging, good and then best architectural practice for a product and the two sets of practices are fundamentally different as they are based upon different principles (e.g. low vs high MTTR, single vs volume instances, designed for purpose vs designed for delivery). A diagrammatic representation of this with respect to computing is ...

So often, the "legacy" can be found with the settlers who have built best architectural practice around consuming their product. This is the importance of theft, the town planners have to steal this activity from the settlers (overcoming any inertia), turn the activity into a more industrialised form (volume operations of good enough designed for delivery e.g. a cloud service for example) and allow for / encourage the development of a new set of architectural practices for consuming this commodity (+utility). In this case what's happening is your Town Planners are killing off the legacy that the Settlers want to hold onto.

Don't confuse Town Planners with the "keepers of the legacy", it just isn't so!

Legacy can incur throughout the structure i.e. there's co-evolution from custom built to product as well and so Pioneers are quite capable of building and maintaining for a very long time a mess of custom built legacy until someone drags it away from them (i.e. the Settlers). I know people talk about bimodal as having one part as managing the "legacy" and the other part managing "digital" but that's just ring fencing part of the organisation from inevitable change and giving them a justification to keep on doing what they were doing (often building data centres!)

I'm not a fan of bimodal, having experienced and been responsible for a dual party structure a decade ago. It creates a host of problems and rather than create a transition usually degenerates into utter warfare. If you're going to split into a "new" camp and "legacy" camp then you're going to be compounding these problems even more. You might as well call it Eloi vs Morlocks.

On a final note, "legacy" isn't something which belongs anywhere. A better description is Toxic IT and it's a debt to the past that is constantly created and needs to be constantly removed.