Tuesday, November 06, 2012

On OpenStack and Dead Ducks ...

I received a message recently that I only referred to Open Stack as a dead duck because it disagreed with my hypothesis on evolution which was unscientific and quasi religious.

This is a very misguided view, so I thought I'd better respond.

On the hypothesis of evolution.

Back between 2005-2007, I collected a reasonable amount of data (around 6,000 data points from telephones to televisions to engineering components to electricity to banking instruments to ... a long list) covering a hundred+ years and discovered a process of how things evolve (as opposed to how things diffuse).  This process which is driven by user and supply competition is described in the diagram below which covers both the correlation and causation of the change.

During 2009-2011, I used a variety of predictions tests to confirm the consequences of the model.  I'm happy to now call this a weak hypothesis being supported by historical data, prediction tests and even published in an article as part of a peer reviewed journal.

Does this mean it is true? Of course not. It means it's a weak hypothesis.  The model describes the journey of any activity (a thing we do) from genesis to custom built examples to product (and rental services) to commodity (and utility services).  Such as the evolution of computing infrastructure from the Z3 to EC2 today (and its equivalents).

Graph of Evolution



For those still in confusion, the above is NOT a diffusion curve (there is no time axis) though it happens to have an S-Curve shape. When the Z3 was created, the act of using computing infrastructure was rare and poorly understand. For Amazon EC2 to appear the act of using computing infrastructure had to be both widespread (ubiquitous) and well understood (certain) in order to support the volume operations that utility provision requires.

Of course both the genesis of an activity and utility provision of a pre-existing activity diffuse but diffusion and evolution are not the same.

The Market Today

When we talk about activities (as opposed to practices and data), we commonly refer to this process of evolution by the term "commoditisation". This is exactly what is happening with computing infrastructure today, it is being commoditised to utility services as exemplified by Amazon EC2.

To counter the hypothesis, you'd have to demonstrate that somehow infrastructure was becoming less of a commodity and that rather than utility services growing that they would suddenly decline and we would return to a world governed by individually bought products.  I have yet to find an example of this throughout history and no reason to suspect that this model will not hold today i.e. utility services for computing infrastructure are here to stay.

I should caveat that there are certain marketing mechanisms and abuses of market power (i.e. oligopoly and reduced competition) which can give an appearance of something "de-commoditising" along with an issue of substitution, but before blindly accepting the opinion that something that has never happened before will now happen, I would ask for one iota of evidence of this.  Simply demonstrate to me that utility services in infrastructure are not growing. Simply explain to me why utility services for infrastructure are not the future and how this transition towards cloud is a mere blip which will soon reverse.  The market says otherwise, history says otherwise and I see no data which supports the alternative opinion. 

Instead I see 6,000 data points plus today's market which says they are wrong.  Now, to call me religious for basing hypothesis on data (both current and past), modelling and prediction tests is farcical. To do so because I won't accept their unsupported, un-evidenced opinion that the entire history of human development is wrong ... well.

The hypothesis states that utility services will grow and dominate computing infrastructure, I see no evidence that this will not happen.

On players

Now, if you happen to agree that the market for computing infrastructure is shifting towards utility services then it becomes a question of who will win that market, will anyone win, what standards will develop and if so how many?

I say standards develop because the natural end state of utility services is provision of fairly standardised components which is commonly through defacto standards followed later by dejeure.  This is an essential part of creating a competitive utility market which also appears common in all utility services.  The answers to these questions depend upon the actions of the players in the market as it forms.

Currently Amazon dominates the infrastructure as a service market and the APIs it provides can be considered defacto.  This is not a permanent situation, it is possible for other players to build a larger ecosystem and to supplant Amazon as the dominant.  An example threat to Amazon may well be Google Compute Engine.

At this moment in time however, Amazon is leading the pack and appears to be visibly growing faster than those around it.  The APIs it provides have been adopted by others (Eucalyptus, CloudStack and even OpenStack) and so whilst this part of the race is not over, it looks like Amazon is a good bet.

On OpenStack

Obviously Amazon has multiple zones and regions but to counter you could attack its weakness of being a single point of failure (a single company) by playing a competitive market game with multiple providers.  In order to do so, you would have to solve the issues of semantic interoperability and this in practice can only be done with an open source reference model.

However, the ecosystem around Amazon provides it with extra-ordinary benefits in terms of innovation, customer focus and efficiency.  Hence a smart play would be to co-opt the ecosystem rather than attempt to build from scratch (i.e. to differentiate away from it). You could differentiate later once your ecosystem was big enough but it seems naive to do this early.

When Rick Clark left Canonical, joined Rackspace and was instrumental in the creation of OpenStack - I fully supported this move of building a competitive marketplace around an open source reference model which co-opted the biggest ecosystem.  However that idea appeared to be short lived as the ideas of differentiation quickly surfaced.

Today, I see considerable problems with OpenStack which I've listed beforehand. My opinion is the project has been hampered by poor leadership, poor strategic play, a lack of major investment by the companies involved, slow development, weak technology and an unnecessary focus on differentiation. It does however have a visible and engaged community.

With the coming price war likely to be initiated between Google Compute Engine vs AWS then OpenStack needs to have in place in the next year, a massive scale competitive market of providers with strong technology.  I hope that they achieve this but I see little evidence of what is needed.  Instead I see further potential issues around a collective prisoner dilemma issue (where members differentiate within the group itself).

So do I believe that in the next year that a benevolent dictator will emerge and resolve these issues, that the technology will become rock solid, that members will invest the billions needed to build a competitive market at scale ... er no.  Which is why I hold the opinion that OpenStack is a wasted opportunity and hence a dead duck. 

So, what if OpenStack fails, will the shift towards utility provision of infrastructure continue? Yes,  well, at least that's the hypothesis.

But what if OpenStack does manage to create a competitive utility market at scale, will the shift towards utility provision of infrastructure continue? Yes, well, at least that's the hypothesis.

This is why the comment that I called OpenStack a dead duck because it "disagreed with my hypothesis on evolution which was unscientific and quasi religious" is misguided whether deliberately so or not. Oh, I'm being too kind ...

The process of evolution is independent of any individual players success or any particular person's opinion. It is simply a consequence of competition.

A final few words 

I realise that people have their own pet opinions (often they call them "theories" when they really shouldn't) and despite scant or more commonly no supporting evidence they argue vociferously that their idea will happen.  If you're one of those then "bully for you", you're obviously omnipotent though that's not the word that comes to mind.

Yes, I have opinions (e.g. OpenStack) which I state clearly as opinions. No-one can predict the interactions of actors in this space, you can only make informed guesses and yes, my opinions are often wrong.  For more on the unpredictability of actors actions then Hayek is worth a read.

Yes, I also have areas of research (e.g. evolution) and this depends upon collection of data, causation, correlation and prediction tests.  Evolution is not time based (i.e. no crystal ball again) and it doesn't depend upon specific actors actions but instead competition between actors.  No, it isn't "right" or "true" or "absolute", it's just the best model I have to explain market phenomenon that are clearly visible for everyone to see.

I would happily dump the model of evolution if someone could finally provide me with a better model and no, I don't count hand wavy concepts with no data, cause, correlation, test or historical examples even if you do believe you're omnipotent.  I'm a skeptic.