Tuesday, July 16, 2013

Could CloudStack, Eucalyptus, Open Nebula and Open Stack all win in the cloud?

I was asked a question whether we could follow a different road and have a market of AWS clones with multiple open source systems providing this and hence all the open source systems win?

Well, assuming we're not talking about the parts of OpenStack fixated on differentiating from Amazon which I have strong opinions on but instead those focused on co-opting Amazon under an embrace (for now) and extend (when your ecosystem is bigger) play then ... technically ... yes. But it's a difficult road.

It has always been possible to create a market of AWS clones with multiple different open source systems providing this. The problem is that in a market of AWS clones then you need to have an open source reference model as the 'standard' for reasons of semantic interoperability. It's easier if that 'reference model' is in fact a system (e.g. Cloud Stack or Eucalyptus) that all the providers implement. This is the more likely route but there is another way.

You could have multiple different open source systems that providers implemented and still maintain semantic interoperability if all those open source systems comply to another 'reference model'. This would actually be beneficial for reasons of competition on operational efficiency and reduction of systemic failures. 

So, surely that means we could have CloudStack, Eucalyptus, Open Nebula and some of the OpenStack party create a rich set of AWS compatible environments? Well, of course but there is a problem. How do you define one thing amongst this group as the 'reference' model?

There is only one way around this that I know and it was a core part of the Zimki plan back in 2005.

For a quick history lesson, Zimki was a Javascript based Platform as a Service, all provided through APIs with object storage systems, billing, management tools, easy switching between Zimki installations etc. Think of Cloud Foundry but way before Cloud Foundry and limited to one language. Zimki was supposed to be open sourced in 2007 in order to establish a competitive market of platform providers but this didn't happen for reasons of the parent company being advised that the future was outsourcing. Long story.

A key part of Zimki was the massive test scripts - everything was accessible and hence testable through APIs. Hence each different installation could be tested and confirmed whether it was compatible (within the bounds of the test script). Think Cloud Foundry Core.

As part of the open sourcing, a remote compatibility testing service was planned with a trademarked image. Any provider meeting the compatibility tests could use the "Zimki Provider" logo rather than the "Powered by Zimki" logo. We had assumed that providers would want to create operational efficiencies but for a market then we wanted to ensure compatibility of services and  switching. Hence the compatibility service (we called it an assurance service) was essential to resolve this.  Zimki also had goals of establishing an exchange and for this the assurance mechanisms were critical. This was all written and talked to death about between 2005-2008, it's old ground.

Now, it is possible to play the same trick with AWS compatibility today. Hence rather than use an open source system as the 'reference model', you could instead create a massive set of tests and a compatibility service. This way you could have multiple open source systems complying to the 'reference model'. I've told enough people enough times in the past that this is a possible route but there's no takers so far. For point of interest, there's potentially huge value in building an AWS compatibility service (assuming we get a market of AWS clones) due to exchanges and other instruments. And yes, you could have multiple compatibility services, in the financial world they would be the equivalent of rating agencies.

Hence these open source groups could come together by creating a massive set of test scripts and providing some form of joint AWS compatibility service. Given a large enough set of test scripts they could ensure reasonable fidelity of behaviour between different open source systems and with AWS. If they did this, then the compatibility service would be the 'reference' model and each provider could show compatibility to it (reinforced with trademark images) and hence you could create a market of AWS clone with multiple open source systems.

So how difficult would this be? Well, given that many of the groups must have some form of AWS test scripts already, then it shouldn't be technically difficult to create a large enough set (think many hundreds of thousands of tests) and establish this. However, you need to bring those groups together to make this happen and that is the tricky part (or it certainly has been in the past).

It would probably require all those interested parties getting together in some form of AWS Compatibility Summit to make it happen and realising this was in their common interest.

But, yes ... it is possible. First of all though, it requires someone to try and make it happen.