Many years ago, at OSCON 2007, I gave a talk on the importance of open sourced standards to create competitive markets in the developing utility computing industry at all levels of the stack. There are many things we require for a functioning market but the most basic is switching which at the very least requires semantic interoperability. To achieve this and to create a free market as opposed to a captured market requires running open source code to be defined as the 'standard'. Any specification will result in interpretation and whilst such translation errors are acceptable in a product world this will not suffice in a utility world.
I've repeated this message many times over the last six years and how "Open source is absolutely essential for creating a competitive free market" but this has to be tempered with a focus on user needs and the reality of the market. Hence, I've long argued for open source AWS clones and the acceptance of EC2 / S3 / EBS as de facto standards. Whether we like it or not, Amazon has a huge ecosystem advantage which cannot be ignored. If you want to create a successful competitive market then embrace rather than differentiate is the order of the day.
Many people tell me that API compatibility is not enough, to which I normally smile sweetly and nod whilst muttering unmentionables and banging my head against a wall. Of course, compatibility with the APIs is not enough because of interpretation errors. For semantic interoperability then systems have to behave in the same way which is also why a reference model (running code) must be the standard and not a piece of paper.
Many people tell me that we will have multiple open source systems in the cloud, to which I normally smile sweetly whilst muttering unmentionables and banging my head against a wall. Of course, you'll get different islands in the cloud but for a market you're going to need semantic interoperability and because of complexity of systems that requires one open source reference model as the standard for that market. You're unlikely (though it's not impossible) to have multiple reference models for the same 'standard' in a market because each will behave differently. Whilst we will end up with some islands, there will be one dominant market based upon the dominant open source reference model providing the de-facto standard as we are fundamentally talking about a commodity market with little differential value. That is likely to translate into one reference model for a large competitive market of AWS clones, one for GCE clones etc.
Many people tell me that OpenStack will win, to which I normally just start muttering unmentionables and banging my head against a wall. There are many potential candidates for example CloudStack, Eucalyptus and even Open Nebula. Key will be the creation of a competitive market and OpenStack already has its own collective prisoner dilemma and questions of interoperability between different distributions of OpenStack. Don't get me wrong, open source is important and a powerful competitive weapon but even more important as an indicator of long term success is the level of strategic play. Mixing both is extremely powerful and there are great players out there such as Netflix. Poor levels of strategic play was why I called into question Apple's future at the height of Apple fever in 2010 when Jobs was still in charge resulting in me taking one of my usual 'cup of tea' bets in 2011. It's the same reason why I call into question the future of OpenStack.
The reference model I favour is CloudStack. First, CloudStack has been clear that it wants to be an AWS compatible cloud (same as Eucalyptus). Second, CloudStack is part of the Apache Software Foundation and backed by a strong player. Third, CloudStack has numerous large scale implementations and most importantly CloudStack shows signs of good game play.
Oh, and before you ask .. don't I advise CloudScaling (an OpenStack shop) and shouldn't I be saying OpenStack over CloudStack? Well, yes I do advise CloudScaling. I also think +Randy Bias is spot on with his focus on making an OpenStack version compatible with AWS because that's what users need. However, when I look at the game being played with CloudStack vs OpenStack vs Eucalyptus vs Open Nebula then I hold the view that CloudStack is playing the better game overall (though I have to admit that it's splitting hairs between CloudStack and Eucalyptus in terms of game play these days).
Oh, and before you ask .. don't I advise CloudScaling (an OpenStack shop) and shouldn't I be saying OpenStack over CloudStack? Well, yes I do advise CloudScaling. I also think +Randy Bias is spot on with his focus on making an OpenStack version compatible with AWS because that's what users need. However, when I look at the game being played with CloudStack vs OpenStack vs Eucalyptus vs Open Nebula then I hold the view that CloudStack is playing the better game overall (though I have to admit that it's splitting hairs between CloudStack and Eucalyptus in terms of game play these days).
Many people tell me that by promoting AWS compatibility that Amazon will win, to which I normally just start banging my head against a wall. If you create a competitive market of multiple providers around a single open source reference model as the standard which is as close to semantically interoperable to AWS as possible then it will grow and you can take advantage of the ecosystem AWS has built. The wider market is looking for second sourcing options.
Once the ecosystem around this open source reference model (i.e. all the providers) becomes bigger than AWS then the open source reference model will start to define the future market. Yes, you will need some form of assurance system (e.g. testing service and trademarked image to provide assurance to end users that a provider is compatible) to prevent a collective prisoner dilemma as companies do like to differentiate on features even though it's clearly not in their interests. Never underestimate the stupidity of companies to undermine their long term future.
You should never confuse AWS (in terms of the APIs and the services that they represent) having won with Amazon having won. In the short to mid-term then yes, it's likely that Amazon will take a huge chunk of the market but that doesn't mean this situation can't be supplanted by a marketplace of AWS clones. What matters is the game play by the individual actors and even Amazon might play a open source card yet. If Amazon does end up dominating the entire market for the long term, well you can put that down to shoddy game play by other executives.
Many people tell me that this is wrong because AWS APIs are proprietary and open APIs will win to which I normally just start thumping with rolled up newspaper at this point. All APIs are principles not expressions and therefore not subject to copyright law in the EU or US. You can't "own" an API. You can certainly be the body that is providing that system (as with Google providing the open source Android) but APIs are inherently open. To wrestle control of an API you need to become the dominant ecosystem. Ignoring the existing dominant ecosystem and user needs in pursuit of an open ideal which doesn't really exist since all APIs are open is ... words fail me.
But what about patents? Have you read any standard documents recently? All standards can be challenged by third party patents. In the world of software, this is a thorn that Governments will finally have to grasp and rebalance the trade to society's advantage. In the UK, judges do a fairly good job of reminding people you can't patent software or software dressed up as something else though lawyers are forever trying to get examples passed them.
Anyway, I'm tired and bored to death of this discussion. Fortunately in the platform space there are good players who understand the game like Cloud Foundry. They get the importance of markets, ecosystems and assurance systems though I do wish they would take Cloud Foundry into the Apache Software Foundation. In the application space, I'm sure we will see others and yes, there's lots and lots of companies and analysts who use the words but don't seem to understand the game.
Whether we want it to happen or not, the cloud market is forming and the 'standards' are already starting to develop in the marketplace. Despite concerns of many this is generally fairly healthy and open source has a critical role to play. Of course, there are many twists and turns yet to come. One huge concern of mine is that governments might waltz in and attempt to interfere with this process.
The UK government in its open standard policy sensibly defined a set of criteria including not imposing undue costs and effective selection requires pragmatic and informed decision making. This is all good, because what you don't want to do is select a 'standard' in a market which is still forming. Standards themselves can, and have, been used as ways of manipulating markets and not necessarily in the interests of end users, though it's often window dressed as such.
For example, let us suppose I was a hardware manufacturer and OpenStack provider who was concerned about the growing strength of AWS and worried that I couldn't differentiate in a commodity market. I would have two choices - I could either compete directly by providing an AWS clone with better price vs QoS or I could find some way of hampering AWS to my own advantage. If I was inclined to take the latter route then I'd probably argue for the adoption of open standards (such as OGF's OCCI) as some form of Government standard under the argument of interoperability. If successful then I'd use this to promote the selling of my OpenStack distribution as compliant to Government standards.
That OpenStack has its own interoperability issues, that OCCI is just a specification not a reference implementation, that user are selecting AWS, that even in open source projects such as Open Nebula you have users demanding more AWS compatibility, that multiple open source projects (Eucalyptus, CloudStack) are focused on AWS compatibility, that the market is too early to define standards and that de facto standards are already emerging through competition would all be neatly sidestepped. This wouldn't be for the benefit of the end users though.
It's always better to let a marketplace mature and form and give it the chance to create its own standards (the de facto) before rubber stamping standards on it. This is also why I have concerns over the whole EC examination into cloud standards. I hope it comes back with a "marketplace is still deciding a list of standards" rather than trying to interfere unnecessarily.
--- update 16th July
I was asked a question whether we could follow a different road and have a market of AWS clones with multiple open source systems providing this?
Well in a market of AWS clones then you need an open source reference model as the 'standard' for reasons of semantic interoperability. Now, you could have multiple open source implementations of that reference model and this would be beneficial for reasons of competition on operational efficiency and reduction of systemic failures. So, surely that means we could have CloudStack, Eucalyptus, Open Nebula and some of the OpenStack party create a rich set of AWS compatible environments?
Well, of course but the problem becomes how do you define one thing amongst this as the 'reference' model and which do you pick? The only way around this that I know is for those groups to come together and create a massive set of test scripts and provide some form of AWS compatibility service. It's not impossible but tricky to do this, however by defining that compatibility service as the 'reference' model then each could show compatibility to it (reinforced with trademark images etc).
It's possible, I've hinted enough times in the past that people could try that route but there's no takers so far.
--- update 16th July
I was asked a question whether we could follow a different road and have a market of AWS clones with multiple open source systems providing this?
Well in a market of AWS clones then you need an open source reference model as the 'standard' for reasons of semantic interoperability. Now, you could have multiple open source implementations of that reference model and this would be beneficial for reasons of competition on operational efficiency and reduction of systemic failures. So, surely that means we could have CloudStack, Eucalyptus, Open Nebula and some of the OpenStack party create a rich set of AWS compatible environments?
Well, of course but the problem becomes how do you define one thing amongst this as the 'reference' model and which do you pick? The only way around this that I know is for those groups to come together and create a massive set of test scripts and provide some form of AWS compatibility service. It's not impossible but tricky to do this, however by defining that compatibility service as the 'reference' model then each could show compatibility to it (reinforced with trademark images etc).
It's possible, I've hinted enough times in the past that people could try that route but there's no takers so far.
