Everything evolves ...

Everything, by which I mean :-

• every activity (what we do)
• every practice (how we do something)
• every mental model (how we make sense of it) 

... evolves from chaotic (poorly understood, rare) to more linear (well understood, commonplace).

This manifests itself in several ways. Hence :-

• For activities we have the evolution from genesis to custom built examples to product (with rental services) to commodity (with utility services).

• For practices we have the evolution from novel to emerging to good to best practice (aka Cynefin framework)

• For data we have the evolution from unmodeled (e.g. we don't know what the structure is and hence we call it unstructured) to modelled.

• For every scientific pursuit we have the evolution from concept to hypothesis to theory to universally accepted.

This process of evolution results in a cycle of change with :-

• Commoditisation of one set of activities enabling the genesis of new higher order activities (componentisation)

• Flows of capital from past to the future higher order systems (creative destruction)

• Increasing pressure (through extraction of higher order value, efficiency and agility) for competitors to keep up with an evolving system (Red Queen Hypothesis)

• Increased consumption of underlying sub systems as they become more commoditised (Jevons Paradox)

• Coevolution of practice with activity. Hence best practice in the product stage <> best practice in the commodity stage. This creates transition cost to cover the technical and architectural debt to the past and is one of the forms of inertia.

• Past success creating inertia to change and acting as the gatekeeper between the economic eras of peace, war and build

• Varying rates of sustaining to disruptive change with the era of war seeing the highest rates of disruptive change due to past industries stuck behind inertia barriers

• New forms of organisation appearing in the war era initiated by evolution of activity that take advantage of the co-evolved practices (e.g. American System, Fordism, Web 2.0)

• The necessity of mapping value chain vs evolution and learning to apply the right strategies dependent upon the state of evolution of a component.

• One size never fits all and the inevitability of arguments such as Agile vs Six Sigman, Push vs Pull, Networked vs Hiearchical due to the consequences of Ashby's Law.

• Exploitation of ecosystems and structure to more effectively cope with the flow from chaotic to linear e.g. ILC or Pioneer-Settler-Town Planner
• NB. When analysing you have to break down value chains into component activities - i.e. you cannot aggregate the mass of activities in a sector to describe an industry as a single activity such as "IT", "Mining" or "Healthcare". Hence IT is not one thing but consists of many component activities which are not the same e.g. Financial ERP, Infrastructure and Social Media. Even these can consist of multiple sub components which need to be broken down. 

Oh, we could go on and on for hours but I won't because I've done this to death many times before over many, many years.

However, I've been asked a couple of times recently about chaotic to linear, so I thought I'd add a number of  graphs which outline the changes and then summarise in the last two. This should be self explanatory to those who know my work. For everyone else, come find me at OSCON or London Cloud Camp at some point in the future where I take time to talk about this within those communities.

First, things diffuse but diffusion is only part of the story. There's a difference between diffusion of a particular phone and the evolution of phones from the first phone to a modern day phone.

Second, the process of evolution is common. To see it you have to first overcome the tendency of mapping with time (the axis you need are ubiquity vs certainty) and second you need to overcome the desire to call everything an innovation. The innovation of the first phone (Genesis), an innovative phone which has some new feature and the innovation of rental services for provision of an existing activity are not the same.

If you don't overcome this you'll just see innovation -> innovation -> innovation -> innovation over a random and unmeasurable time pattern rather than genesis -> custom built -> product -> commodity over defined and measurable axis.

This process is unavoidable because it's driven by user and supply competition.

Third, commoditisation and not genesis of activities cause a rapid change in society. It's the commoditisation of an activity that enable higher order systems to appear (nut + bolt -> machine, electricity -> computing etc). The new higher order systems are the new sources of wealth (hence we get creative destruction and capital flows) and the old "new" thing now becomes cost of doing business, standardised and more automated.

Hence as we evolve, we move up (and create) higher orders of our value chain.

It's worth noting that the net effect of commoditisation is not only efficiency but increased rates of higher order system creation and agility (componentisation) and the ability of companies to extract wealth from this. This combination of effects result in what is known as the Red Queen i.e. you have no choice but to evolve in order to retain a relative position compared to a surrounding ecosystem that is evolving.

Fourth, it's not just activities that evolve but practices (from novel, emerging, good to best). Practices are how we do something, activities are what we do. Now practice can co-evolve with activities i.e. best practice in the product world is not the same as best practice in a commodity or utility world. This change of practice is one of the many sources of inertia for company due to the cost of transition (which can also be thought of as technical or architectural debt).

Fifth, this entire process can be represented as a cycle of change with the commoditisation of pre-existing activities enabling the genesis of higher order system which in turn commoditise. Inertia (both user and provider) acts as an important gatekeeper and separates three economic eras of peace (relative competition), war (fight for survival) and growth (time of wonder).

Each of these eras have different modes of "innovation" i.e. the peace era, the time of highest margin, is all about sustaining change (which tends to exceed disruptive).

In the war era (caused by commoditisation of an activity due to its ubiquity and certainty achieving a state that this is possible i.e. volume operations of standard components) the disruptive change exceeds sustaining.

The growth era which is subsequently propelled by this, is a time of great experiments, of wondrous change due to the formation of these higher order systems which are by nature uncertain (see evolution graph near the top).

Sixth, this entire pattern repeats ad nausea - from the industrial age to the age of machines to the age of electricity to the internet age. What we see is commoditisation of a pre-existing activity initiates a state of war where new organisations form taking advantage of new co-evolved practices and these new organisations disrupt past industries stuck in peace mentality behind inertia barriers.

During this period where we focus on efficiency, automation and standardisation we also see an explosion of growth of higher order systems which we can't predict (electricity led to radio, hollywood etc), a shift of capital and the formation of new giants. And so it continues ...  Oh, and yes ... each time we see explosions of unmodelled data and endless arguments over classification etc.

Seventh, this occurs but at a macro scale (i.e. ages or k-waves if you prefer) and at a local scale (specific industry). Cloud is an example of this and has resulted in new forms of organisation (the new Fordism) as did every other example of this cycle (American System, Fordism, Web 2.0 etc etc)

The new Fordism practice will diffuse just like the old Fordism practice did. So, I've added a table of what you probably look like today and what you will look like tomorrow.

Seventh, when it comes to strategy, in order to effectively play the game you need to understand the landscape (situational awareness). 

The landscape is your value chain (i.e. what you do and the component which make what you do possible) and evolution stage of those components. The reason you need both is that the properties of activities, practices etc change as they evolve (more on that next).

If you don't see the landscape then any strategy you play whether an ecosystem game or open source or anything else ... is hit and miss, a shot in the dark. You could easily commoditise a barrier to entry into your industry or fail to effectively manage a threat to your value chain.

Lastly, as the activities (or practices or data etc) evolve they moves through three different stages of chaotic, transitional and linear. The characteristics of each stage is different and polar opposite at the extremes. This is why one size never fits all - whether it's project management method, structure, customer focus ... the whole lot. 

This is also why we end up with tiresome arguments over extremes such as agile vs six sigma, push vs pull marketing, networked vs hierarchical, NoSQL vs SQL, listen to vs don't listen to customers ... when the actual answer is always both.

Ok, this brings you upto about 2008 in terms of core models with some extra and more modern goodness in terms of how organisations change and the relationship between economic ages.

Don't ask me for papers on this, I don't write public papers, nor do I publish the thousands of data points I've collected. Why? Because I have too much fun using this stuff against others.

Do feel free to help yourself to anything you find useful. Be skeptical, be very skeptical. It's nothing more than weak hypothesis (i.e. concept, data, causation, correlation and prediction testing).

A magic quadrant of cloud ...

Everything you need to know about infrastructure as a service in one magic quadrant of EGO vs ECO system ...

Adoption cycles

I couldn't stop laughing at @jdrumgoole's tweet on IT departments adoption on new technology.

So, I had to create a graphical representation of the adoption cycle.

PS. I've been asked by several people whether they can use this graph in presentations / posts. You are welcome to do whatever you wish with it. It is provided CC BY SA 3.0 and to the extent possible under law, I waive any right to attribution.

PPS. The above graph is purely for humour and it's not based upon any actual measurement. It does however seem to have touched a raw nerve with some people.

This is not my country ... is it?

Being English, there are many ideals that I believe my country should aspire to from freedom, equality, openness and meritocracy to a deep sense of fraternity. However, I increasingly feel this is not my country but some of social engineering experiment gone horribly wrong. I happen to blame Milton Friedman and his acolytes but that's another story, another day.

I could labour on about the appalling state of social mobility in the UK, the reduction in civil liberties, the inequality of austerity measures when people like Jimmy Carr try to dodge taxes or the banker's scam over LIBOR for financial gain or even to the lack of social cohesion as we all slavishly follow the tune of TINA (there is no alternative) and submit ourselves to a "dog eat dog" mentality. Of course, dogs don't actually eat dogs but what's a bit of reality to the quasi scientific cult of monetarism that rules our financial institutions.

I thought I'd however discuss a more human story than LIBOR. A tale of an elderly couple.

One was a drunk and eventually it killed this former World War II member of the Polish Air Force. According to him, he had escaped Germany, walked across Europe, fought in the war alongside the British in the Polish Air Force and lost most of his family in concentration camps (brothers, sisters, mother, father, aunts and uncles - you get the picture). I never had reason to doubt him. I grew up on these stories just in case you wonder why I despise fascism so much.

He had settled in the UK, and with his wife they had worked hard all their lives (he as a self employed painter, she worked for British Gas). He often claimed to have painted Margaret Thatcher's house near Dulwich. Eventually, the couple retired to their council home with a small pension and almost no savings.

They never made a great deal of money and they could never afford to own a home. The one exception was when they bought their council home (under a government scheme) and then promptly sold it to a property agent. This netted them a small sum (£30,000 or so) but they were relatively hard up and it cost them their home and their community.

Before you judge, the practice of property agents buying up ex-council homes and financing this was rife. Have you never wondered how those expensive ex-council properties in London were cleared of the previous tenants? A bit of money to the hard up is a great way of creating an exodus. In general, the people who lost the most were the previous tenants - they lost community, friends and homes for a bit of cash.

So, they ended up in a new council estate having to play all sorts of dodgy games because the council wasn't supposed to give them a home as they could afford to buy a home (which they couldn't). They knew no-one, they were isolated and they already had a tendency to drink but they told everyone they were fine. This became the norm. A bottle of cheap special offer discounted supermarket whiskey for breakfast ... sounds the ticket.

Of course, their health deteriorated and a couple of trips to the hospital ensued for numerous conditions - strokes, emphysema, suspected heart attacks - you name it. The care was modest, the nurses marvellous but it usually involved some sort of merry go around about how quickly they could be got rid of. The hospital wanted them out, social services didn't want anything to do with them and they suggested the grand idea that one of their daughters - either the one who has a mental health condition or the other one who is in and out of hospital with cancer and chemotherapy - could look after two drunk octogenarians with medical complications who wanted to stay in their council home. The wider family offered to help but this was rebuffed. So despite going into a home being the sensible option - the two were sent back to their council property. Brilliant.

So, further hospital trips ensued and the family decided it was best if home help was hired (because no-one lived near them) and conveniently one was available who was apparently certified by social services and works as a home help for others. The rest of the family kept in contact with the sisters The sisters also mainly kept in contact with the couple by telephone because relationships were fraught ... a combination of trying to deal with two cantankerous old drunks with severe short term memory and other health issues is only made more complex when there's bad blood due to the past. I would occasionally visit and despite knowing them when I was younger, they would often not know who I was when I turned up. My personal relationship became fraught, my visits became very infrequent and a year or two would slip by without me noticing.

And so the situation continued for a decade in a state of limbo with occasional visits by the sisters until the elderly gentleman (their father) died. The council house the couple lived in had degenerated in the end into a bit of a pigsty. The home help had helped herself to the entire of the £30,000 of the savings the couple had and that appears to be about all the helping that was really happening. The police were called, the home help had vanished. The house had also been recently fleeced of anything of value. What has actually happened here, no-one actually knows ... getting a coherent story from the surviving drunk who can't remember who you are or that they've spoken to you thirty minutes ago is practically impossible.

A few days later, the surviving member of this couple was found rambling incoherently drunk in the middle of a large town at midnight and managed to put herself back in hospital. I know, I was called by one of the sisters and I offered help. It wasn't needed as the social services said mental health would look after her. Alas mental health said social services will deal with her. The hospital was just trying to get rid of her.

Naturally, they were trying to get the daughters (i.e. one who has ongoing cancer, the other who is schizophrenic) to look after her - anything to avoid putting her into a home. The willingness of social services to save money by not providing care is as equally ferocious as the willingness of some Gov departments to waste vast sums of money on pointless IT projects and mega buck management consultants. We really need to rethink our priorities.

In between this saga of social care, I read about more austerity, scumbags like Jimmy Carr trying to dodge taxes and the Bankers (who've already had loads of our taxpayers cash) scamming LIBOR.

At times like this, in my darkest moments, I think privately to myself "I want to see public flogging brought back". This tells me, something is going seriously wrong for me to get so angry with the way people behave to each other that I can think such awful thoughts. But mostly, the anger is from a feeling of guilt - I should have done more, if only I had visited more often I would have known, if I hadn't just listened and investigated, I should have been more observant, I was too willing to leave it to the sisters, too eager to take the rebuff, I had allowed that relationship to become fraught etc. Too little, too late.

This tale is small fry compared to the suffering that goes on around the world but it's important to me. That elderly Polish gentleman, who escaped a concentration camp, lost most of his family within them, walked across Europe, fought for the RAF and lived the majority of his adult life in Britain ... that was my grandfather.

It makes me feel guilty for my life choices. I was smart, I could have taken that job trading at the bank, I could have scammed LIBOR and all the little people, earned the big money, stuffed my money in offshore bank accounts and then I could have had the riches to do something about this. Their lives could have been different. Except, there's always going to be someone else's grandfather.

We need to build a more caring and loving society.