Interesting moves by VMware

Many years ago I made a prediction that VMware (or more specifically its master EMC) would focus on an open source platform play and look to monetize its existing hypervisor based business.

The latter part I suspected will occur through an IPO of VCE or some equivalent move. The reason being is that at this moment much of the buzz in financial analyst circles is around hybrid cloud environments consisting of a combination of private & public infrastructure and VMware's technology is touted as well placed for this industry.

What the analysts don't seem to understand is that this hybrid model is purely transitional and will be replaced in the majority by a hybrid model of public and public i.e. use of multiple public sources. The private enterprise cloud market is at best transitional and has a limited lifespan until it becomes niche.

Still, there's nothing like selling at the top of the market an area of technology which you know will ultimately be disrupted. If the analysts help you do this, bully for you.

However, EMC is very astute. There's an awful lot of value which can be gained by providing a commodity infrastructure services through the development of a wide ecosystem and the exploitation of such through an ILC (innovate-leverage-commoditise) model.

So, I've often toyed with the idea that VMware (EMC) would :-

1.  Develop and launch on an open source platform play. This is a key part for the future in terms of developing a competitive market of providers, destroying any competitor's ability to differentiate in this space and enabling VMware to exploit this space with a range of higher order services (see management tools).

2.  IPO or sell the existing hypervisor technology. This is to maximimize the value of a range of technologies which will ultimately be disrupted and provides a neat way of overcoming inertia to change within the organisation by spinning it out of the company.

3.  Develop and launch a range of cross public cloud management tools. This would be part of a future plan of creating brokerage, exchange and assurance capabilities across both infrastructure and platform and monetizing these layers of the computing stack.

4.  Launch a major open source IaaS offering.

My thoughts on this ...

So, the first part and the development of an open source platform play occurred a couple of years after I made the prediction. From all reports, Cloud Foundry is making great progress and there are multiple providers being set up from ActiveState, Uhuru Software, PaaS.io, Tier 3, AppFog and VMware's own service.

The second part of the prediction I consider likely to happen in the near future and I view that VCE is the likely vehicle for this. It's critical that any IPO is achieved prior to any market decline and the subsequent questioning over the future of private enterprise class clouds. It's not that they're not viable, it's just you have to run them with a brutal commodity focus to make them operationally efficient.

I've seen examples of private clouds operating at a third the price that AWS charges publicly on a like by like comparison of default EC2 instances. Which is why I'm one of those who argues that AWS is probably operating at a very high margin (>60%) and there's a lot of scope for price reductions.

I believe there is a reason why AWS doesn't simply reduce its public pricing drastically and that's because I suspect that AWS has a bottleneck - buying up enough land to build data centres to keep up with its rapidly expanding growth. Dropping prices aggressively will just increased demand and exacerbate this problem. Any price reductions will have to be carefully managed.

This is also why I think that the GCE (Google Compute Engine) focus on initiating a price war is smart because whenever you face a competitor with a bottleneck, creating a price war to increase demand beyond their ability to supply will naturally fragment the market.

I said the same to IBM, Dell and HP back in 2009 (when I was with Canonical) and told them it was in their interests to move quickly and create a price war with Amazon unless they wanted to end up being in a strategically weak supplier position to a few dominate cloud players. I don't think any of them listened but that's not my problem, that's theirs.

Anyway, to operate at the levels to be operationally efficient it is going to be difficult if not impossible when you use proprietary licensed software. My models reckon you can afford to spend a couple of dollars (i.e. $1-3) per year per EC2 equivalent on software licensing & software maintenance. We are told that vCloud Director is "non-existent" in the service providers space, this doesn't surprise me.

I don't see a big future for these software products in the public service provider space and I only see a time limited future in the private enterprise space before it becomes niche. Hence IPO before the trouble hits is a smart play in my book and EMC does tend to act in a really smart fashion.

However, this leaves two others parts of the prediction which I've not made public before especially as the open source play I've considered to be too bold a move even for EMC. I'm now coming to the view that I've underestimated their boldness.

My view here is fairly simple. Once an IPO had been achieved (or some other mechanism of separation) then a fully fledged IaaS attack could occur. I've been watching this space with interest looking for hints that such a change might be coming which is why I'm increasingly confident that this is a possible game they might play.

First, I noted that VMware / EMC joined the Open Compute project which if you're going to get into the game of building massive scale and efficient commodity based data centres is where I'd start.

Next, I noted that Paul Maritz has been promoted up to running strategy for EMC itself, which in my view is a shoe-in eventually for the CEO role.

Then, I noted that VMware acquired Nicira.

Finally, I heard about project Zephyr and how VMware (i.e EMC) was willing to attack the IaaS space. This was enough to make me publish this post.

Now, if I was going to play a major open source IaaS game then I'd probably base it around Cloud Stack (Citrix's donation of $200m worth of cloud technology to the ASF) and combine this with an S3 equivalent and software networking capabilities. For the S3 equivalent I would probably acquire Basho for Riak CS and then add this into the Cloud Stack project under ASF.

If I then had within the company the following skills - hypervisor technology, large scale efficient data centres (i.e. open compute), distributed storage (Riak CS), software defined networking (Nicira), management tools (Dynamic Ops) then I'd be almost ready to play a major open source IaaS play.

The other bits I'd need are some partnerships on highly commoditised hardware (I'll note the recent partnership between EMC and Lenovo), acquisition of large data centre space (Project Zephyr) and skills with Cloud Stack (which VMware already seems to have).

Of course, Project Zephyr will be touted as selling existing technology but if I was running this play then that would be just be marketing cover for the hidden agenda. And to be truly Machiavellian, given that there would be an inevitable reaction of existing partners to Project Zephyr, then I'd use this as marketing cover for the reasons why I'd IPO VCE - an argument of separation of concerns.

So in my play, into VCE would go the existing hypervisor business (along with all the inertia it creates to change) and then this would be IPO'd in order to "avoid the channel conflict" that Zephyr creates with partners and Nicira creates with Cisco. In one swoop I'd have maximised value (through IPO) on a business that I think will be disrupted in the medium term and kept all the components that I think are the long term future by using a completely plausible story of channel conflict. The capital I'd have raised on an IPO would be ploughed right back into an open source IaaS play which I would then reveal.

At lot of what is needed to make this happen appears to be already in place, so as completely far fetched as it sounds, I'm expecting in under 18 months for EMC / VMware to have acquired Basho, launched an open source IaaS effort and IPO'd VCE.

Of course, I have no inside knowledge of EMC / VMware strategy. Their plan maybe something else but this is what I would do. If if happens and it succeeds then this will be a truly outstanding example of how to deal with inertia and the threat of disruption.

[Update 15 Mar 2013]

First, VMware is pushing heavily on the hybrid model which is all par for the course and the open source platform plays have been made and pushed into a new spinoff known as Pivotal Initiative (under Paul)

Second, EMC itself appears to be scouting out some more commodity IaaS providers which makes perfect sense given the above.

In my original version, EMC would have pushed the "to be disrupted technology" into VCE and flogged it off at the top of market whilst keeping the interesting future technology within VMware. The excuse for flogging VCE off would be a channel conflict because VMware would make a big commodity IaaS play.

However, given that EMC has pushed the interesting future technology into the Pivotal Initiative, leaving the "to be disrupted technology" inside VMware then VMware itself can be flogged off (i.e. you don't have to use VCE to create the separation).

However, you still need the plausible excuse of channel conflict and a way of making a big commodity IaaS play. Which is why I think the EMC interest sounds reasonable.

So rather than:-

• "to be disrupted technology" into VCE (EMC exit)
• "interesting future technology" in VMware (EMC keep)
• VMware to play a commodity IaaS play  (EMC keep)
• VCE ("to be disrupted technology") is then flogged off (EMC exit)

An alternative route would be:-

• "to be disrupted technology" in VMware (EMC exit)
• "interesting future technology" into Pivotal Initiative (EMC keep)
• EMC to play a commodity IaaS play (EMC keep)
• VMware  ("to be disrupted technology") is then flogged off (EMC exit)

The effect is the same, EMC keeps the interesting tech (open source platform play), builds a commodity IaaS play and flogs off hypervisor technology for a good price. I was a bit surprised that DynamicOps etc remained within VMware, this seems a curious decision.

Will it pan out like this? We shall see. Whilst overall trends (e.g. evolution from product to utility) are highly predictable, the actions of individual actors aren't.

Self disruption and super linear ...

On Self Disruption

Back around 2005, in the days of Fotango, we had started to introduce a structure which I commonly refer to a Pioneer, Settler and Town Planner. This is structure which is not by type (i.e. IT, Marketing, Business Development) but instead by Flow (from chaotic to linear, from genesis of an activity to commodity).

Under the structure, Pioneers are rewarded for experimentation and the genesis of new activities. Settlers steal successful patterns from the Pioneers or any wider ecosystem to create more defined components, a library of useful things. The Town Planners steals successful components from the Settlers to create commodity components through utility services with a focus on automation and standardisation.

The advantage of the structure is that each group feeds of the previous with Settlers stealing from Pioneers and Town Planners stealing from Settlers. The circle is completed because the Pioneers build on the services the Town Planners create.

Now back in the days of Fotango we started with IT and called these groups development, framework and operations however we had already started the process of folding business development and marketing into this structure. I know of three other companies that use such a structure, alas it's not enough to test the findings conclusively.

That said, the early findings included a remarkable surge in efficiency and rate of genesis of new activities. This combined with our creation of a platform underpinning this concept led to some stunning results including the development and delivery of a prototype wiki with client side preview from concept to live on the web in under 53 minutes.

However, the real beauty of the model is that it's self disruptive due to the structure creating a pressure for each group to steal from each previous group. The creation of such a pressure overcomes inertia which may have formed through past success.

On Super Linear

Our focus at Fotango was also on the development of ecosystems hence our entire platform was provided though "open" APIs back in 2006. We had several years of experience in building with web services and we intended to open source all the components in 2007 to create a competitive market with a view of establishing higher order systems such as exchanges, brokerages and assurance.

Alas, there's another story here of political capital and inertia within large companies and since this is well documented, I won't go there again.

The model which I use to describe the goal of an ecosystem approach is known as Innovate-Leverage-Commoditise and its use is rather simple and becoming more common today.  First, you start by providing more linear activities i.e. those suitable for utility provision because they're widespread and relatively well understood. Examples include compute resource, storage, databases etc.

The purpose of these is to allow an ecosystem of other companies to build on top of your service and to encourage those companies to innovate (as in genesis of new activities) by reducing their cost of failure (through provision of utility services). In effect, some of those companies you can consider to be an extension of your own Pioneers.

As any new activities spreads through the ecosystem, you should be able to detect this change in consumption of APIs. This is the Leverage component of the model as you use the ecosystem itself to do the pattern spotting and it in effect operates as an extension of your own Settlers.

One a pattern is spotted, you commoditise this to utility services. Yes, you will be accused of eating the ecosystem, though you can always control this messaging by balancing acquisition vs copying. However, each time you eat the ecosystem, the new component services you create should help grow it. This balancing act is critical to keeping the ecosystem healthy.

As a company, you will appear to be highly innovative (but in reality others are creating those new activities), highly customer focused (but in reality the ecosystem is telling you what you need) and highly efficient (because you're focused on utility services).

Bye, bye the old mantra of choose one of these strategies. That's been a busted flush for many years.

Now, as a very weak hypothesis the rate of innovation (genesis), customer focus and efficiency can increase almost linearly with the size of the ecosystem, though the volume of data is not strong enough to publish this (unless we're talking about various popular management publication in which case any old nonsense seems to get through).

The beauty of this is that ecosystem growth can be super linear with the actual size of the company and hence if the hypothesis holds then it's possible to become super linear for innovation, customer focus and efficiency through an ecosystem model. Given what I know, this appears to be the case but it's not conclusively shown.

Whilst competitors have built self contained Towers, Amazon has built a city with itself at the heart of it. The old model of big companies are less nimble is a busted flush here because genesis of activities and agility extend into the surrounding ecosystem.

On Self Disruption and Super Linear.

It's perfectly possible to organise a company using an ecosystem model such Innovate - Leverage - Commoditise around a structure of Pioneer - Settler - Town Planner.

In this case the Pioneers works on encouraging and creating new activities - a mix with a heavy emphasis on Community Evangelists.

The Settlers work on identifying the new patterns in the ecosystem and exploiting this - a mix with a heavy emphasis on Data Scientists, Strategists and Business Development.

The Town Planners work on commoditising new patterns - a mix with a heavy emphasis on Engineers and Operations.

There is no reason that I'm aware of why a company can't be both continually self disruptive and superlinear for innovation, customer focus and efficiency. In fact, the only reasons which I know that prevent this is the poor use of ecosystem (if at all) and poor structure (e.g. organisation by type).

I suspect that Amazon will in the future be cited in endless "management" books on why these problems can be solved. This is also why I've said many times over the years that Amazon is likely to be that $1 trillion market cap company and it will become far more powerful the bigger it gets.

Why the Rant?

I'm tired of people telling me that you can either be innovative or customer focused or efficient but you can't be all of them.

I'm tired of people telling me that companies can't be super linear for the above and twisting Geoffrey West's work as evidence of this.

I'm tired of people telling me that companies can't be self disrupting and that it's inevitable that companies will be disrupted.

The above statements only hold if you build your company with management practices from the last Century.

Open your eyes, the software world is already starting to operate in an environment where all of the above is possible and this will soon extend out into manufacturing (though commoditisation of the manufacturing process).

Get used to it and start thinking about how to do this better than competitors.

Rant over ...

Everything evolves ...

Everything, by which I mean :-

• every activity (what we do)
• every practice (how we do something)
• every mental model (how we make sense of it) 

... evolves from chaotic (poorly understood, rare) to more linear (well understood, commonplace).

This manifests itself in several ways. Hence :-

• For activities we have the evolution from genesis to custom built examples to product (with rental services) to commodity (with utility services).

• For practices we have the evolution from novel to emerging to good to best practice (aka Cynefin framework)

• For data we have the evolution from unmodeled (e.g. we don't know what the structure is and hence we call it unstructured) to modelled.

• For every scientific pursuit we have the evolution from concept to hypothesis to theory to universally accepted.

This process of evolution results in a cycle of change with :-

• Commoditisation of one set of activities enabling the genesis of new higher order activities (componentisation)

• Flows of capital from past to the future higher order systems (creative destruction)

• Increasing pressure (through extraction of higher order value, efficiency and agility) for competitors to keep up with an evolving system (Red Queen Hypothesis)

• Increased consumption of underlying sub systems as they become more commoditised (Jevons Paradox)

• Coevolution of practice with activity. Hence best practice in the product stage <> best practice in the commodity stage. This creates transition cost to cover the technical and architectural debt to the past and is one of the forms of inertia.

• Past success creating inertia to change and acting as the gatekeeper between the economic eras of peace, war and build

• Varying rates of sustaining to disruptive change with the era of war seeing the highest rates of disruptive change due to past industries stuck behind inertia barriers

• New forms of organisation appearing in the war era initiated by evolution of activity that take advantage of the co-evolved practices (e.g. American System, Fordism, Web 2.0)

• The necessity of mapping value chain vs evolution and learning to apply the right strategies dependent upon the state of evolution of a component.

• One size never fits all and the inevitability of arguments such as Agile vs Six Sigman, Push vs Pull, Networked vs Hiearchical due to the consequences of Ashby's Law.

• Exploitation of ecosystems and structure to more effectively cope with the flow from chaotic to linear e.g. ILC or Pioneer-Settler-Town Planner
• NB. When analysing you have to break down value chains into component activities - i.e. you cannot aggregate the mass of activities in a sector to describe an industry as a single activity such as "IT", "Mining" or "Healthcare". Hence IT is not one thing but consists of many component activities which are not the same e.g. Financial ERP, Infrastructure and Social Media. Even these can consist of multiple sub components which need to be broken down. 

Oh, we could go on and on for hours but I won't because I've done this to death many times before over many, many years.

However, I've been asked a couple of times recently about chaotic to linear, so I thought I'd add a number of  graphs which outline the changes and then summarise in the last two. This should be self explanatory to those who know my work. For everyone else, come find me at OSCON or London Cloud Camp at some point in the future where I take time to talk about this within those communities.

First, things diffuse but diffusion is only part of the story. There's a difference between diffusion of a particular phone and the evolution of phones from the first phone to a modern day phone.

Second, the process of evolution is common. To see it you have to first overcome the tendency of mapping with time (the axis you need are ubiquity vs certainty) and second you need to overcome the desire to call everything an innovation. The innovation of the first phone (Genesis), an innovative phone which has some new feature and the innovation of rental services for provision of an existing activity are not the same.

If you don't overcome this you'll just see innovation -> innovation -> innovation -> innovation over a random and unmeasurable time pattern rather than genesis -> custom built -> product -> commodity over defined and measurable axis.

This process is unavoidable because it's driven by user and supply competition.

Third, commoditisation and not genesis of activities cause a rapid change in society. It's the commoditisation of an activity that enable higher order systems to appear (nut + bolt -> machine, electricity -> computing etc). The new higher order systems are the new sources of wealth (hence we get creative destruction and capital flows) and the old "new" thing now becomes cost of doing business, standardised and more automated.

Hence as we evolve, we move up (and create) higher orders of our value chain.

It's worth noting that the net effect of commoditisation is not only efficiency but increased rates of higher order system creation and agility (componentisation) and the ability of companies to extract wealth from this. This combination of effects result in what is known as the Red Queen i.e. you have no choice but to evolve in order to retain a relative position compared to a surrounding ecosystem that is evolving.

Fourth, it's not just activities that evolve but practices (from novel, emerging, good to best). Practices are how we do something, activities are what we do. Now practice can co-evolve with activities i.e. best practice in the product world is not the same as best practice in a commodity or utility world. This change of practice is one of the many sources of inertia for company due to the cost of transition (which can also be thought of as technical or architectural debt).

Fifth, this entire process can be represented as a cycle of change with the commoditisation of pre-existing activities enabling the genesis of higher order system which in turn commoditise. Inertia (both user and provider) acts as an important gatekeeper and separates three economic eras of peace (relative competition), war (fight for survival) and growth (time of wonder).

Each of these eras have different modes of "innovation" i.e. the peace era, the time of highest margin, is all about sustaining change (which tends to exceed disruptive).

In the war era (caused by commoditisation of an activity due to its ubiquity and certainty achieving a state that this is possible i.e. volume operations of standard components) the disruptive change exceeds sustaining.

The growth era which is subsequently propelled by this, is a time of great experiments, of wondrous change due to the formation of these higher order systems which are by nature uncertain (see evolution graph near the top).

Sixth, this entire pattern repeats ad nausea - from the industrial age to the age of machines to the age of electricity to the internet age. What we see is commoditisation of a pre-existing activity initiates a state of war where new organisations form taking advantage of new co-evolved practices and these new organisations disrupt past industries stuck in peace mentality behind inertia barriers.

During this period where we focus on efficiency, automation and standardisation we also see an explosion of growth of higher order systems which we can't predict (electricity led to radio, hollywood etc), a shift of capital and the formation of new giants. And so it continues ...  Oh, and yes ... each time we see explosions of unmodelled data and endless arguments over classification etc.

Seventh, this occurs but at a macro scale (i.e. ages or k-waves if you prefer) and at a local scale (specific industry). Cloud is an example of this and has resulted in new forms of organisation (the new Fordism) as did every other example of this cycle (American System, Fordism, Web 2.0 etc etc)

The new Fordism practice will diffuse just like the old Fordism practice did. So, I've added a table of what you probably look like today and what you will look like tomorrow.

Seventh, when it comes to strategy, in order to effectively play the game you need to understand the landscape (situational awareness). 

The landscape is your value chain (i.e. what you do and the component which make what you do possible) and evolution stage of those components. The reason you need both is that the properties of activities, practices etc change as they evolve (more on that next).

If you don't see the landscape then any strategy you play whether an ecosystem game or open source or anything else ... is hit and miss, a shot in the dark. You could easily commoditise a barrier to entry into your industry or fail to effectively manage a threat to your value chain.

Lastly, as the activities (or practices or data etc) evolve they moves through three different stages of chaotic, transitional and linear. The characteristics of each stage is different and polar opposite at the extremes. This is why one size never fits all - whether it's project management method, structure, customer focus ... the whole lot. 

This is also why we end up with tiresome arguments over extremes such as agile vs six sigma, push vs pull marketing, networked vs hierarchical, NoSQL vs SQL, listen to vs don't listen to customers ... when the actual answer is always both.

Ok, this brings you upto about 2008 in terms of core models with some extra and more modern goodness in terms of how organisations change and the relationship between economic ages.

Don't ask me for papers on this, I don't write public papers, nor do I publish the thousands of data points I've collected. Why? Because I have too much fun using this stuff against others.

Do feel free to help yourself to anything you find useful. Be skeptical, be very skeptical. It's nothing more than weak hypothesis (i.e. concept, data, causation, correlation and prediction testing).

A magic quadrant of cloud ...

Everything you need to know about infrastructure as a service in one magic quadrant of EGO vs ECO system ...

Adoption cycles

I couldn't stop laughing at @jdrumgoole's tweet on IT departments adoption on new technology.

So, I had to create a graphical representation of the adoption cycle.

PS. I've been asked by several people whether they can use this graph in presentations / posts. You are welcome to do whatever you wish with it. It is provided CC BY SA 3.0 and to the extent possible under law, I waive any right to attribution.

PPS. The above graph is purely for humour and it's not based upon any actual measurement. It does however seem to have touched a raw nerve with some people.

This is not my country ... is it?

Being English, there are many ideals that I believe my country should aspire to from freedom, equality, openness and meritocracy to a deep sense of fraternity. However, I increasingly feel this is not my country but some of social engineering experiment gone horribly wrong. I happen to blame Milton Friedman and his acolytes but that's another story, another day.

I could labour on about the appalling state of social mobility in the UK, the reduction in civil liberties, the inequality of austerity measures when people like Jimmy Carr try to dodge taxes or the banker's scam over LIBOR for financial gain or even to the lack of social cohesion as we all slavishly follow the tune of TINA (there is no alternative) and submit ourselves to a "dog eat dog" mentality. Of course, dogs don't actually eat dogs but what's a bit of reality to the quasi scientific cult of monetarism that rules our financial institutions.

I thought I'd however discuss a more human story than LIBOR. A tale of an elderly couple.

One was a drunk and eventually it killed this former World War II member of the Polish Air Force. According to him, he had escaped Germany, walked across Europe, fought in the war alongside the British in the Polish Air Force and lost most of his family in concentration camps (brothers, sisters, mother, father, aunts and uncles - you get the picture). I never had reason to doubt him. I grew up on these stories just in case you wonder why I despise fascism so much.

He had settled in the UK, and with his wife they had worked hard all their lives (he as a self employed painter, she worked for British Gas). He often claimed to have painted Margaret Thatcher's house near Dulwich. Eventually, the couple retired to their council home with a small pension and almost no savings.

They never made a great deal of money and they could never afford to own a home. The one exception was when they bought their council home (under a government scheme) and then promptly sold it to a property agent. This netted them a small sum (£30,000 or so) but they were relatively hard up and it cost them their home and their community.

Before you judge, the practice of property agents buying up ex-council homes and financing this was rife. Have you never wondered how those expensive ex-council properties in London were cleared of the previous tenants? A bit of money to the hard up is a great way of creating an exodus. In general, the people who lost the most were the previous tenants - they lost community, friends and homes for a bit of cash.

So, they ended up in a new council estate having to play all sorts of dodgy games because the council wasn't supposed to give them a home as they could afford to buy a home (which they couldn't). They knew no-one, they were isolated and they already had a tendency to drink but they told everyone they were fine. This became the norm. A bottle of cheap special offer discounted supermarket whiskey for breakfast ... sounds the ticket.

Of course, their health deteriorated and a couple of trips to the hospital ensued for numerous conditions - strokes, emphysema, suspected heart attacks - you name it. The care was modest, the nurses marvellous but it usually involved some sort of merry go around about how quickly they could be got rid of. The hospital wanted them out, social services didn't want anything to do with them and they suggested the grand idea that one of their daughters - either the one who has a mental health condition or the other one who is in and out of hospital with cancer and chemotherapy - could look after two drunk octogenarians with medical complications who wanted to stay in their council home. The wider family offered to help but this was rebuffed. So despite going into a home being the sensible option - the two were sent back to their council property. Brilliant.

So, further hospital trips ensued and the family decided it was best if home help was hired (because no-one lived near them) and conveniently one was available who was apparently certified by social services and works as a home help for others. The rest of the family kept in contact with the sisters The sisters also mainly kept in contact with the couple by telephone because relationships were fraught ... a combination of trying to deal with two cantankerous old drunks with severe short term memory and other health issues is only made more complex when there's bad blood due to the past. I would occasionally visit and despite knowing them when I was younger, they would often not know who I was when I turned up. My personal relationship became fraught, my visits became very infrequent and a year or two would slip by without me noticing.

And so the situation continued for a decade in a state of limbo with occasional visits by the sisters until the elderly gentleman (their father) died. The council house the couple lived in had degenerated in the end into a bit of a pigsty. The home help had helped herself to the entire of the £30,000 of the savings the couple had and that appears to be about all the helping that was really happening. The police were called, the home help had vanished. The house had also been recently fleeced of anything of value. What has actually happened here, no-one actually knows ... getting a coherent story from the surviving drunk who can't remember who you are or that they've spoken to you thirty minutes ago is practically impossible.

A few days later, the surviving member of this couple was found rambling incoherently drunk in the middle of a large town at midnight and managed to put herself back in hospital. I know, I was called by one of the sisters and I offered help. It wasn't needed as the social services said mental health would look after her. Alas mental health said social services will deal with her. The hospital was just trying to get rid of her.

Naturally, they were trying to get the daughters (i.e. one who has ongoing cancer, the other who is schizophrenic) to look after her - anything to avoid putting her into a home. The willingness of social services to save money by not providing care is as equally ferocious as the willingness of some Gov departments to waste vast sums of money on pointless IT projects and mega buck management consultants. We really need to rethink our priorities.

In between this saga of social care, I read about more austerity, scumbags like Jimmy Carr trying to dodge taxes and the Bankers (who've already had loads of our taxpayers cash) scamming LIBOR.

At times like this, in my darkest moments, I think privately to myself "I want to see public flogging brought back". This tells me, something is going seriously wrong for me to get so angry with the way people behave to each other that I can think such awful thoughts. But mostly, the anger is from a feeling of guilt - I should have done more, if only I had visited more often I would have known, if I hadn't just listened and investigated, I should have been more observant, I was too willing to leave it to the sisters, too eager to take the rebuff, I had allowed that relationship to become fraught etc. Too little, too late.

This tale is small fry compared to the suffering that goes on around the world but it's important to me. That elderly Polish gentleman, who escaped a concentration camp, lost most of his family within them, walked across Europe, fought for the RAF and lived the majority of his adult life in Britain ... that was my grandfather.

It makes me feel guilty for my life choices. I was smart, I could have taken that job trading at the bank, I could have scammed LIBOR and all the little people, earned the big money, stuffed my money in offshore bank accounts and then I could have had the riches to do something about this. Their lives could have been different. Except, there's always going to be someone else's grandfather.

We need to build a more caring and loving society.

In Search of Excellence ... revisited

On the Nov 1st, 1982, Tom Peters and Robert Waterman published "In Search of Excellence". This book has become a cornerstone of management reading.

However, it's based upon case studies of 62 companies and identifying those that are excellent according to a particular framework. It doesn't in my view explain the underlying processes of change and hence I'd argue that it's flawed, fundamentally.

The problem with the work, which is common to many management literature, is that it's based fundamentally on case studies with no underlying model of explanation, no cause, no correlation and no predictive tests. The worst examples of this are those books which run on an assumption that big companies know what they're doing and hence case studies on big companies are some guide to how you should operate. That's fairly delusional given any cursory examination of corporate history.

Despite this, the book has some good themes and I thought it would be interesting to view where these companies have got to in the last 30 yrs, a sort of where are they now?

I'll publish the list on the 1st Nov but I thought I'd start with a draft list, needs lots of checking, confirmation etc. I'll update this post over the months, at the moment it's rough notes.

If someone would like to do an actual study of the "In Search of Excellence 62", set-up a wiki etc - then I'd love to hear about this.

It would probably make a good book and fascinating reading. I'd suggest you publish it on the 1st Nov (hint, hint) and I'd certainly buy it at a reasonable price.

Company	1982	2012	Original
Allen-Bradley	Private	Still in operation as a brand-name for a line of Factory Automation Equipment. Acquired by Rockwell Automation in 1985.	Defunct
Amdahl	Public	Defunct 1997. Bought by Fujitsu	Defunct
Digital Equipment	Public	Defunct 1998. Bought by Compaq who was bought by HP.	Defunct
Emerson Electric	Rank 118	Ranked 120 in the Fortune 500. No change.	Operating
Gould	Private	Acquired in 1988 by Nippon Mining, still in operation.	Operating
Hewlett Packard	Rank 110	Rank 11 in the Fortune 500. Significant Increase.	Operating
IBM	Rank 8	Rank 18 in the Fortune 500. Fall.	Operating
NCR	Public	Acquired by AT&T in 1991. A restructured spin-off in 1996 was given the same name.	Defunct
Rockwell	Rank 48	Divisions sold off in the 1990s and in 2001 Rockwell International was split into two companies, Rockwell Automation and Rockwell Collins ending the run of what had once been a massive and diverse conglomerate. Rockwell Automation ranks at 466, Rockwell Collins at 478. Significant fall.	Operating
Schlumberger	$39.38	Still in operation. NYSE $59.67.	Operating
Texas Instruments	Rank 91	Rank 175 in Fortune 500. Significant Fall.	Operating
United Technologies	Rank 20	Rank 44 in Fortune 500. Fall.	Operating
Western Electric	Public	Split up into several divisions within AT&T with parts being spun off. Defunct in 1995.	Defunct
Westinghouse	Rank 34	Split up into several divisions and sold off with the core of the company renamed to CBS Corporation (Rank 174). Brand name of Westinghouse still continues today. Significant Fall	Operating
Xerox	Rank 42	Rank 121 in the Fortune 500.Significant Fall	Operating
Blue Bell	Public	Acquired by VF Corporation in 1986.	Defunct
Eastman Kodak	Rank 28	Filed for Chapter 11 Bankruptcy in 2012. Significant Fall	Operating
Atari	Rank 483 in 1988	Acquired by Hasbro in 1998. Infogame acquired Hasbro Interactive in 2000. Final assets of Atari acquired in 2008 for $11 million total. Atari suffers from significant financial issues. Significant Fall	Operating

On Competing with APIs

APIs are just a vehicle for creating ecosystems and it's through exploitation of ecosystems (under models such as innovate-leverage-commoditise) that a company can aim to become linear for innovation, customer focus and efficiency with the growth of the ecosystem.

So when competing in an active market on APIs providing similar activities (such as Infrastructure), you really only need to ask yourself - what are the relative sizes of the active developer ecosystems consuming those APIs?

1. If yours is bigger and growing faster relative to competitors … you're winning. If done correctly, you're likely to become even more innovative, customer focused and efficient than your competitors. Whoot!

2. If yours is smaller but growing faster relative to competitors … you should eventually win. It might take you many years and you run the risk of the larger competitor finding that killer API but if things keep on as is, then that's fine.

3. If yours is smaller and growing slower relative to competitors … then copy their APIs. This is especially true if what we're talking about is a commodity (i.e. we're discussing utility services rather than rental services for a product) because attempts to differentiate on a commodity are likely to be futile - you'll probably end up in a niche area.

In the latter case, it's far better to tag along by co-opting the dominant ecosystem with a view of extending it once you have found a way of making your ecosystem dominant. For example, if you're OpenStack then one advantage you can create over Amazon is by creating a competitive market of OpenStack providers. Hence by adopting the dominant EC2/S3/EBS APIs, your focus should be to co-opt the existing ecosystem and offer it an advantage through a marketplace of providers.

Alas, achieving this would mean giving up on ideas about differentiating around a commodity or providing an on-ramp private cloud targeted towards your own public service. It would also require those involved overcoming their own iterative prisoner's dilema where an individual company attempts to differentiate to their own advantage weakening the system as a whole.

Whether OpenStack or Eucalyptus or CloudStack will achieve this and become dominant, well it all depends upon how well they play the game. At this moment in time, it certainly appears as if Amazon is just running away with it.

I said to myself I would never again write another post about bloody APIs. This subject has bored the industry and myself rigid for the last five years. I retired from Cloud over two years ago to get away from this nonsense on differentiating a commodity and ... well nothing surprises me about cloud. The industry will probably still be arguing over this in five years from now. Fortunately at least Eucalyptus and CloudStack understand things clearly, along with Canonical / Ubuntu and Mark Shuttleworth

Also, before anyone says it ... APIs are of course not all that's necessary for semantic interoperability, it's a starting point. Cloning APIs only get you so far but at least it's further than not cloning APIs.

Also, before anyone says it ... won't copying APIs just force providers into a price war and a spiral down to lowest margin. Absolutely and if you're a hardware provider (e.g. Dell, HP, IBM) then that's exactly what you want.

Why? Well if the market continues as is then you're going to find yourself in a strategically weak position with a single large buyer - Amazon. What any hardware vendor should be looking for is a fragmented market. By causing a price war, you'll increase demand and that's good.

Why? Because Amazon appears to run at high margin (if you compare to the best examples of commodity private cloud). Now Amazon is a shrewd player and so you can assume that the reason why they've been dropping prices but not to the extent possible is because they're controlling demand which seems to be growing at somewhere near a whopping 200%. You can therefore assume that Amazon has a bottleneck - I'd hazard a guess at buying land for data centres. So, if you force a rapid growth in demand by causing a price war then you will naturally fragment the market. This is good for the strategic position of hardware suppliers.

Also, before anyone says it ... yes, that will only work if computing infrastructure is highly elastic which the last thirty years of data says it is.

In general when it comes to APIs, I'm secretly hoping that Amazon will open source the technology behind EC2 / S3 / EBS in order to shut up the rest of the industry. I don't expect them to actually do so (for numerous reasons) but it would great for my own benefit if Amazon just ended these tiresome discussions.

STRATAfication of society

First, I'm going to start this post by saying I'm a great fan of the O'Reilly Strata conferences and the surge of interest in data science. Now, I'm going to argue that people really haven't thought through the consequences of this.

Before I begin some background information. 

One of the consequences of the development of utility services in computing is the growth of ecosystem models such as innovate, leverage and commoditise or ILC (see Graph 5). These models appear to allow a company to become linear (possibly super linear) for both customer responsiveness and the innovation of new activities, with respect to ecosystem size. Combined with economies of scale, the effect of this is that those companies which exploit such models will become harder to compete with as their ecosystems grow. Contrary to Porter's three strategies, these companies can become simultaneously more efficient, more innovative and more customer focused with growth.

The trick to this is data, the timeliness of the data and the ability to respond. Use of APIs inherently provides more timely and more voluminous data than market research on product usage  but it's the combination of API provision to an ecosystem (provides access to raw information on consumption), metrics on usage and diffusion (algorithms), utility infrastructure (raw compute power) and distributed big data systems (processing) which makes exploitation of this more possible. 

Ecosystem models such as ILC where innovation is encouraged elsewhere through reducing cost of failure, the ecosystem is then leveraged to spot  diffusion of innovation and then those diffusing activities are commoditised to component services in order to grow the ecosystem - a process of eating the ecosystem in order to grow it - have only become possible in the last decade. 

The practice should itself diffuse to dominate all industries touched by technology and those thinking that a bigger company (in terms of size) can beat a smaller company with a larger ecosystem (in terms of size) should prepare themselves for a shock. It's not companies but ecosystems that fight in this future world.

This won't be the last time we see billion dollar market cap companies employing thirteen people, quite the opposite. We should expect this, because whilst we're on the tail end of one industrial age which has covered the internet, social media and cloud computing, we're also at the start of a new age which will be built on many of the components that the last produced.

These component will include utility infrastructure (i.e. compute power and storage), big data processing and increasingly a widespread sensor network from smart phones to the software components that we interact with such as SIRI and EVI. 

These components will lead to agent software, an environment where my abilities are augmented by the network around me. From dynamically determining my schedule, to working out what to buy my sister for her birthday to giving me a heads up on my bosses latest meeting. For some background on this see "Any Given Tuesday"

However, the ability of my agent network will depend upon access to information, algorithms, raw compute power and processing capabilities. Even if access to information and processing capabilities were open (as in open data and open source respectively), my ability to exploit this will depend upon the algorithms and volume of computer power I can afford.

In other words could my ability to exploit agents to my personal advantage relative to over others will depend upon how much I can afford?

So not only is education a resource to be bought, so too will the ability to create advantage through agents and to hence more effectively exploit the precious time we all have. Counter to the levelling that open source has brought, this change could lead to an increasingly stratified world between the have more and the have less.

As Tim O'Reilly has pointed out, we are part of this machine and we're not yet sure what we are building. In the excitement of all the possibilities of big data, there exists a dystopian nightmare of reduced social mobility. 

Could my failure to afford the right sort of education and the right sort of agent condemn my son's future to answering endless Turk requests on "which would make a better birthday present?" in order to serve the software agents of others who spend their time on higher matters because their parents can afford to buy them into this strata? Does it matter? Will ability matter? 

Certainly the movement of social mobility in the US / UK gives me pause for thought but who is looking into the social impacts of big data?

Terms I'm using ...

Thought I'd write a note on the general terms that I use.

• Activities are what we do.
• Activities evolve through discrete states - genesis, custom built, product & rental services, commodity & utility services.
• Practices are how we do it.
• Practice evolve through discrete states - novel, emerging, good and best.
• The evolution of an activity can cause the co-evolution of a practice i.e. the shift of computing infrastructure from from product to utility services has caused the development and evolution of a new set of architectural practices.
• Both activity and practice states can be characterised into common phases which have common properties - chaotic, transitional and linear.
• For activities these phases are
• Genesis is chaotic
• Custom, Product & Rental Services are transitional
• Commodity & utility services are linear.
• For practices these phases are
• Novel is chaotic
• Emerging and Good are transitional
• Best is linear.
• The largest inertia barriers to change are found in the evolution of activities from the transitional to the linear phase i.e. the shift from product & rental services to commodity & utility services.
• Inertia barriers are instrumental in controlling the shift between different economic eras - known as the peace, war and build era.
• The eras make up a repeating cycle of change which maybe specific to an industry or have wider economic effect.
• The largest of these cycles of change are known as Ages.

Pioneers, Settlers and Town Planners

I recently heard of a third organisation which has a pioneer, settler and town planner (PST) like structure rather than the usual organisation by type - IT, Finance, Marketing - or the various derivations of (Geography & Type, Business Unit & Type).

I was going to do a long rambling post on evolution, how activities and practices move from chaotic (poorly understood, uncertain, constantly changing, rare, future source of worth)  to more linear (well defined, predictable, stable, common, cost of doing business) and how organisations contain a mass of these activities and practices. Understanding this and using the right methods and tactics is important to creating a balance between the unstable but potentially high margin activities (chaotic) and the stable and low margin (linear).

This process of evolution from chaotic to linear is why "one size fits all" mentalities are so dangerous because you either impact survival today (through poor efficiency) or survival tomorrow (through future wealth creation). This creates the badly termed "innovation paradox" of Salaman and Storey.

There is however a route by which you can create a profitable but stable organisation which deals with the constant cycle of change, except unfortunately traditional organisational structures (by type) get in the way. They do so by obscuring the natural evolution (or flow) of activities from chaotic to linear which is driven by user and supply competition. Invariably the traditional structures built on type result in alignment issues and a host of other problems.

More cell based structures (e.g. Two Pizza) appear better than traditional structures at dealing with these issues but as I accidentally found out in 2004 (but couldn't explain why back then) this can be enhanced by a pioneer, settler and town planner (PST) structure which appears to solve the problem of flow enabling high rates of innovation of new activities and efficiency continuously.  These days, I can explain why this should work but with so few examples then it could just be coincidence or some other bias.

Under PST, there is no IT, Finance or Marketing departments or any grouping by type. There is only a structure defined by evolution and flow - hence pioneers, settlers and town planners.

Now, I could go through the details in a long rambling post but I've done this countless times before in presentations around the world over many, many years. So, instead I'll just add a few diagrams (pinched from various presentations) on PST which is useful for those with a basic understanding of evolution, value chains and the flow of chaotic to linear (for some background see Ten Graphs on Organizational Warfare)

Figure 1 - How things evolve.

Figure 2 - Mapping an organisation, value chain vs evolution

Figure 3 - Structure around it.

Figure 4 - An organisation based on theft.

Overall, I'm happy to hear of another example and I've also been told of a possible fourth. Alas, I'll need several hundred more operating under many years of economic competition before I can actually demonstrate the difference has significance.

The models say this is a more "right" structure than by type and if successful even in these few cases, the practice should diffuse. Time will tell and I'll be watching those companies with a great deal of interest.

20th March 2013

Updated images with higher resolution screenshots with less Neapolitan Ice Cream effect.

The interesting thing about cutting costs

I've been reading about RIM's strategy and the announcements by Chief executive Thorsten Heins that they plan to build new smart phone software in an industry being increasingly commoditised through Android. They also plan layoffs to make RIM more profitable. 

I'm not impressed. First some background, just in case you need it.

All business activities (what we do) evolve through a common pathway but this also leads to co-evolution of practice (how we do it). For example computer infrastructure has evolved from its modern day genesis with the Z3 to custom built systems, products with rental services and finally to more commodity with utility services. As the activity has evolved, so has architectural practices for scaling, system resilience and overall fault tolerance. In the product world, best architectural practice was scale-up with bigger machines, N+1 and disaster recovery tests. In the utility world, this is increasingly being replaced with the emerging practices of distributed system, design for failure and chaos engines. These emerging practices will ultimately become best practice for the utility world.

The evolution of activity and practice both follow a path from a chaotic phase (rare, constantly changing, unmeasurable, poorly understood) to a more linear phase (common, stable, measurable and well defined). In activities this is shown as genesis to custom to product to commodity. With practices you have novel to emerging to good to best practice.

One of the consequence of this process of change is that higher order systems built with best practice in one evolutionary phase (such as applications built on infrastructure with architectural practices of scale up and N+1) incur a transitional cost to the new phase (i.e. you cannot simply port applications based upon scaleable and highly resilient hardware to a utility world offering volume provision of good enough and more standardised machines).

This is one of many inertia barriers to change but despite their existence you have to transition for reasons of competition. I've covered this many many times before, so I'll simply say it's not a question of if but when.

Inertia is important in the cycle of change (the process of commoditisation actually enables genesis of new higher order systems which then commoditise themselves) because it acts as a gatekeeper between different economic states or eras of the cycle - peace, war and build.

The peace era is one of relative competition, where sustaining change exceeds disruptive change, where listening to customers, product improvement and a focus on profit / margin are critical.

The war era is one that is often initiated by a new entrant not encumbered by past business models and the inertia to change this creates. It's a time where disruptive change exceeds sustaining, where competition is a fight for survival, where customers have their own inertia and say they want things (e.g. better SLAs, better hardware) before abandoning this to adopt volumes of good enough. It is a time of commoditisation and the disruption of past industries, past skills and past roles.

Overlapping this is the build era, a time of wonder where capital quickly flows from past industries to new. In this time of creative destruction, new organisations appear exploiting the new co-evolved practice and we see explosions in the rate of creation of higher order systems. It's a time of increased experimentation and uncertainty where there are no customers to listen to and massive increases in unmodeled data with endless arguments over classification. Eventually we see the formation of bubbles and corrections and the new super giants which will settle down to dominate the new industries as they enter a peace era and wait for the next wave.

Now different parts of the economy and its value chains can be in different evolutionary phases at the same time, so whilst one industry is in a era of peace, another is in a era of war. Whilst this cycle can be local it can bubble up through combinations of activities that are evolving in close tine order to create macro economic waves which we call Kondratiev waves or Ages.

The underlying drivers of this (user and supply competition), the process of evolution, co-evolution of practices,  the cycle it creates, the new forms of organisation, the increases in data and the link to macro economic waves can be traced back over 200 years. In business, I've been actively using these models and making public presentations on them over the last 6 years. The entire system not only has cause and correlation (thousands of data points) but is most importantly predictive - something which I finally was able to test at the LEF last year.

So, to the title - the problem with cutting costs.

If your industry (i.e. the parts of value chain which you sell) are in a peace era then cutting costs through efficiency to increase profitability due to declining revenue can be a good play, assuming you don't reduce barriers to entry into the space. There are many reasons why you would do this and often you can clear out a lot of waste in the organisation.

However, if your industry has moved into the war then then cutting costs through staff to restore profitability due to declining revenue is often a terrible move. The problem is your revenue is eroding due to a change in the value chain and the commoditisation of the activity. You need to respond by adapting and possibly moving up the value chain. However, by layoffs you're likely to get rid of those people who were seen to be less successful in the previous era.  That doesn't sound too bad but the result is you end up with a higher density of people successful in the past models (which are now in decline due to evolution) and hence you'll tend to increase your cultural inertia to change. Revenue will continue to drop and you'll start a death spiral. 

What you of course should be doing is adapting and realising that the tactics you play in one era are not the same as another (peace vs war etc). Now any large organisations has multiple different values chains in different evolutionary phases and you have to see this and know how to switch context between them in order to choose the right tactics. Naturally, most people don't manage to achieve this which is why big companies often die but at least that keeps things interesting.

So back to RIM. Trying to produce a new software product in an increasingly commoditised industry seems to smack of inertia to change. Cutting staff through layoffs will almost certainly reinforce this. RIMs future in my view has become pretty bleak. So expect the usual patent spats of a wounded creature which is in a self inflicted death spiral. Also, expect the usual nonsense about how it wasn't the strategy but the culture, market and competitors that finished RIM. Alas, this is a classic case of a succession of CEOs out of their depth and driving a good company to the wall.