Tuesday, November 06, 2012

On OpenStack and Dead Ducks ...

I received a message recently that I only referred to Open Stack as a dead duck because it disagreed with my hypothesis on evolution which was unscientific and quasi religious.

This is a very misguided view, so I thought I'd better respond.

On the hypothesis of evolution.

Back between 2005-2007, I collected a reasonable amount of data (around 6,000 data points from telephones to televisions to engineering components to electricity to banking instruments to ... a long list) covering a hundred+ years and discovered a process of how things evolve (as opposed to how things diffuse).  This process which is driven by user and supply competition is described in the diagram below which covers both the correlation and causation of the change.

During 2009-2011, I used a variety of predictions tests to confirm the consequences of the model.  I'm happy to now call this a weak hypothesis being supported by historical data, prediction tests and even published in an article as part of a peer reviewed journal.

Does this mean it is true? Of course not. It means it's a weak hypothesis.  The model describes the journey of any activity (a thing we do) from genesis to custom built examples to product (and rental services) to commodity (and utility services).  Such as the evolution of computing infrastructure from the Z3 to EC2 today (and its equivalents).

Graph of Evolution



For those still in confusion, the above is NOT a diffusion curve (there is no time axis) though it happens to have an S-Curve shape. When the Z3 was created, the act of using computing infrastructure was rare and poorly understand. For Amazon EC2 to appear the act of using computing infrastructure had to be both widespread (ubiquitous) and well understood (certain) in order to support the volume operations that utility provision requires.

Of course both the genesis of an activity and utility provision of a pre-existing activity diffuse but diffusion and evolution are not the same.

The Market Today

When we talk about activities (as opposed to practices and data), we commonly refer to this process of evolution by the term "commoditisation". This is exactly what is happening with computing infrastructure today, it is being commoditised to utility services as exemplified by Amazon EC2.

To counter the hypothesis, you'd have to demonstrate that somehow infrastructure was becoming less of a commodity and that rather than utility services growing that they would suddenly decline and we would return to a world governed by individually bought products.  I have yet to find an example of this throughout history and no reason to suspect that this model will not hold today i.e. utility services for computing infrastructure are here to stay.

I should caveat that there are certain marketing mechanisms and abuses of market power (i.e. oligopoly and reduced competition) which can give an appearance of something "de-commoditising" along with an issue of substitution, but before blindly accepting the opinion that something that has never happened before will now happen, I would ask for one iota of evidence of this.  Simply demonstrate to me that utility services in infrastructure are not growing. Simply explain to me why utility services for infrastructure are not the future and how this transition towards cloud is a mere blip which will soon reverse.  The market says otherwise, history says otherwise and I see no data which supports the alternative opinion. 

Instead I see 6,000 data points plus today's market which says they are wrong.  Now, to call me religious for basing hypothesis on data (both current and past), modelling and prediction tests is farcical. To do so because I won't accept their unsupported, un-evidenced opinion that the entire history of human development is wrong ... well.

The hypothesis states that utility services will grow and dominate computing infrastructure, I see no evidence that this will not happen.

On players

Now, if you happen to agree that the market for computing infrastructure is shifting towards utility services then it becomes a question of who will win that market, will anyone win, what standards will develop and if so how many?

I say standards develop because the natural end state of utility services is provision of fairly standardised components which is commonly through defacto standards followed later by dejeure.  This is an essential part of creating a competitive utility market which also appears common in all utility services.  The answers to these questions depend upon the actions of the players in the market as it forms.

Currently Amazon dominates the infrastructure as a service market and the APIs it provides can be considered defacto.  This is not a permanent situation, it is possible for other players to build a larger ecosystem and to supplant Amazon as the dominant.  An example threat to Amazon may well be Google Compute Engine.

At this moment in time however, Amazon is leading the pack and appears to be visibly growing faster than those around it.  The APIs it provides have been adopted by others (Eucalyptus, CloudStack and even OpenStack) and so whilst this part of the race is not over, it looks like Amazon is a good bet.

On OpenStack

Obviously Amazon has multiple zones and regions but to counter you could attack its weakness of being a single point of failure (a single company) by playing a competitive market game with multiple providers.  In order to do so, you would have to solve the issues of semantic interoperability and this in practice can only be done with an open source reference model.

However, the ecosystem around Amazon provides it with extra-ordinary benefits in terms of innovation, customer focus and efficiency.  Hence a smart play would be to co-opt the ecosystem rather than attempt to build from scratch (i.e. to differentiate away from it). You could differentiate later once your ecosystem was big enough but it seems naive to do this early.

When Rick Clark left Canonical, joined Rackspace and was instrumental in the creation of OpenStack - I fully supported this move of building a competitive marketplace around an open source reference model which co-opted the biggest ecosystem.  However that idea appeared to be short lived as the ideas of differentiation quickly surfaced.

Today, I see considerable problems with OpenStack which I've listed beforehand. My opinion is the project has been hampered by poor leadership, poor strategic play, a lack of major investment by the companies involved, slow development, weak technology and an unnecessary focus on differentiation. It does however have a visible and engaged community.

With the coming price war likely to be initiated between Google Compute Engine vs AWS then OpenStack needs to have in place in the next year, a massive scale competitive market of providers with strong technology.  I hope that they achieve this but I see little evidence of what is needed.  Instead I see further potential issues around a collective prisoner dilemma issue (where members differentiate within the group itself).

So do I believe that in the next year that a benevolent dictator will emerge and resolve these issues, that the technology will become rock solid, that members will invest the billions needed to build a competitive market at scale ... er no.  Which is why I hold the opinion that OpenStack is a wasted opportunity and hence a dead duck. 

So, what if OpenStack fails, will the shift towards utility provision of infrastructure continue? Yes,  well, at least that's the hypothesis.

But what if OpenStack does manage to create a competitive utility market at scale, will the shift towards utility provision of infrastructure continue? Yes, well, at least that's the hypothesis.

This is why the comment that I called OpenStack a dead duck because it "disagreed with my hypothesis on evolution which was unscientific and quasi religious" is misguided whether deliberately so or not. Oh, I'm being too kind ...

The process of evolution is independent of any individual players success or any particular person's opinion. It is simply a consequence of competition.

A final few words 

I realise that people have their own pet opinions (often they call them "theories" when they really shouldn't) and despite scant or more commonly no supporting evidence they argue vociferously that their idea will happen.  If you're one of those then "bully for you", you're obviously omnipotent though that's not the word that comes to mind.

Yes, I have opinions (e.g. OpenStack) which I state clearly as opinions. No-one can predict the interactions of actors in this space, you can only make informed guesses and yes, my opinions are often wrong.  For more on the unpredictability of actors actions then Hayek is worth a read.

Yes, I also have areas of research (e.g. evolution) and this depends upon collection of data, causation, correlation and prediction tests.  Evolution is not time based (i.e. no crystal ball again) and it doesn't depend upon specific actors actions but instead competition between actors.  No, it isn't "right" or "true" or "absolute", it's just the best model I have to explain market phenomenon that are clearly visible for everyone to see.

I would happily dump the model of evolution if someone could finally provide me with a better model and no, I don't count hand wavy concepts with no data, cause, correlation, test or historical examples even if you do believe you're omnipotent.  I'm a skeptic.

6 comments:

Russ Nelson said...

That's too bad, because about a year ago I handed off to Wikipedia a project which uses OpenStack for bulk storage. They're now using it for all their big media files.

I had proposed an alternative solution which was much closer in line to what they already had, but which would scale MUCH larger. They rejected it because "Then we would be responsible for maintaining it." They may yet get their wish. :)

cal said...

Nice thought provoking analyses. On the "next year that a benevolent dictator emerges" point though, does the data of how organizational governance emerge support this hypothesis? I can't think of communities with benevolent dictators where they dictator wasn't dictator from early inception, admittedly when i rack my brains my own data set is thin (linux, php, python, perl, moodle). I'm not sure that dictators are likely to emerge post project inception. If OpenStack was to have a BDFL would they not have one by now?

Simon Wardley said...

Hi Russ,

There is plenty of historical data for the process of evolution which itself depends upon competition and not individual actions. The rise of IaaS today is just another example and another data point.

The process of evolution cannot however be predicted across a time basis because that pre-supposes knowledge of individual actors. Also the question of whether one actor or not we will win is largely opinion and dependant not only upon their actions but the actions of others (see Hayek and the difficult of modelling economic behaviours of actors).

Hence whether OpenStack will succeed or not is a matter of opinion, as no-one has a crystal ball.

I don't happen to subscribe to the view that it will succeed because of :-
a) the rapid growth of AWS and its strong ecosystem
b) the potential price war between AWS vs GCE
c) the issue of a collective prisoner dilemma with differentiation between the "distros"
d) the visible lack of major scale investment from the members (by major I mean $ billions)

In its favour is has a wide range of companies involved.

Simon Wardley said...

Hi Cal,

The most common property of successful open source projects is not a particular license (GPLv3 vs Apache2.0), or whether the process of development is open or not (see Apache vs Android) or ... a long list.

The most common property is the existence of a benevolent dictator at the early stages of its history. The key part is of course benevolence.

Now, whether that dictator can emerge ... who knows.

The Community Blogger said...

Simon:

I think this is an excellent post and something that all OpenStack community members should read and think about. However, I expect most to just rant and rave that you are a fool and don't see how OpenStack is already the standard. (How can you be a standard with just a few deployments?)

I was at the Summit a few weeks back and there is so much internal discussion of how great they are that they are now believing their own marketing. A very dangerous place for a community to be in.

The foundation and community is still spending too much time on how great they are, how large they are and not enough on who the customers are and what scale they are seeing. Without proof, the community is just a marketing engine with nothing behind it.

I am also in favor of them having a BD as they are now 2.5 years old and are still not ready for prime time. How can this be? Easy - the corporate members care about their version of OpenStack but not the core project itself and without a true technical leader in place to control the product features from corporations the product will not make it to market in time. The community has had enough time to see the current process is not getting it done and a change is needed. Doubt me? Well, 2.5 years is an ETERNITY in software development time and for a cloud I have yet to hear of a deployment beyond 500/5000 nodes- that is not scale. Where are the 50,000+ node deployments?

Neill Turner said...

I quite like the Openstack API because it allows metadata so different public clouds can add their own bits of data. The problem with Amazon AWS API as a cloud standard is that it is not extensible.

For people writing tools and clients for clouds it is much easier to build for openstack and just handle the differences between clouds by handling metadata differences that coding for totally different APIs.

So I think the OpenStack API is the best contender to become the Linux of cloud APIs even if some of the companies who participate don't become successful cloud providers.

Maybe Google Compute Engine will add the openstack API to their public cloud in the future?