On Standards and Open Source
The issue of standards and in particular open standards is a hotly debated topic. The recent UK Government consultation on open standards was embroiled in much of the politics on the subject with even a media expose of the chair of a consultation meeting as a member of a paid lobbyist group. The rumours and accusations of ballot stuffing of ISO meetings with regards to Microsoft’s OOXML adoption as an open standard are also fairly widespread. The subject matter is also littered with confusing terms from FRAND (fair, reasonable and non discriminatory licensing) being promoted as an “open” standard despite it being IP encumbered by definition.
In general, the principle of standards is about interoperability. In practice, it’s appears more of a battleground for control of a developing market. Standards themselves can also create new barriers to entry into a market due to any onerous cost of implementation. There are also 17 different definitions of what an “open standard” is varying from international bodies to governments. Of these the OSI definition is probably the most respected. Here, open standards are defined as those which have no intentional secrets, not restricted by patents or other technology, no dependency on execution of a license agreement, freely and publicly available and royalty free.
When I talk about evolution of systems (whether activities, practices and data) these are all about meeting specific needs such as providing a web site or carbon dioxide emissions of a country. However standards refer to generic needs that apply across many stages of evolution – the need for things to work together and the need to be able to switch between solutions. These generic needs change in implementation as the underlying system evolves.
For example, in the world of software products then standards can provide interoperability between products and hence the potential to replace one product with another. The standard is normally articulated as a principle of working (i.e. how data is transferred and interpreted) in a document and the expression of this standard (i.e. the code itself) is left up to the product vendor.
This process is often imperfect as one vendor’s interpretation and expression of a principle might not match another’s. However since the switching between solutions is not usually time critical for the end user then some level of imperfection is acceptable i.e. you own the product and if you decide to switch you can migrate at your own pace. Though it should be noted that the process of migration is often a fraught one due to the imperfections.
In the world of utility services, the switching time can be critical and immediate such as the termination of a service. Here imperfections in migration are extremely undesirable and effective switching requires semantic interoperability between the services. In other words, any code and data is understood in the same way between the providers. In practice this can only be achieved if both providers are running the same system or strictly conforming to a reference model (an expression of code) rather than their interpretation of a documented principle.
Switching is an essential part of any competitive market and with utility services then the principle in a standards document is not enough and a reference model (i.e. running code, the expression) is required. If that market is to be unconstrained (i.e. free) then that expression needs to be open itself.
Hence open standards in the product world can simply be open documented principles but in the utility world they require open source reference models (i.e. running code). This simple fact is counter to how standards have been used in the past battles between products and hence the confusion, debates and arguments over the terms are unsurprising.
On Clouds, Open Source and Standards
So, we come to the cloud, which is simply the evolution of a range of IT activities from a product and product rental model to one of commodity and utility services. If we are going to see competitive free markets in this space, then open source reference models are essential. But the value to the end user is the market, not whether this system is more open or not.
An example of this is in the IaaS space. If you ask some of the big users of AWS what they want, they often reply with multiple AWS clones and rarely with another IaaS with another API. To create a marketplace of AWS clones you're going to need to start with an open source system that provides the EC2/S3/EBS APIs with multiple implementations (i.e. providers of such).
I first raised this issue at Web 2.0 in 2006 and by 2007 the whole debate of "open APIs" was kicking off and it has raged ever since. The argument today goes that EC2/S3/EBS are not "open". However APIs are simply principles, they cannot be owned only your expression can be. This has been re-affirmed in several court cases over the last few years.
This debate has already started to switch to "But the process of creating EC2/S3/EBS isn't open" ... well neither is the process for the development of Android and the end user doesn't care. As Benjamin Black once said "Solve user problems or become irrelevant".
The user problem is a competitive free market of multiple utility providers offering the defacto standard (which in case you haven't realised is EC2/S3/EBS) and not a plethora of multiple APIs basically doing the same thing, with super APIs attempting to manage this mess under various cries of being more "Open".
Matt Asay does a good job hitting the point home with his post of "Whose cloud is the open sourciest ... who cares?". Let us be crystal clear here, Matt isn't against open source in the cloud, he (like most of us) understands its essential importance. However, Matt also understands that what needs to be focused on is the user need for competitive markets.
Open source is absolutely essential for creating a competitive free market but the focus should be on solving users need i.e. creating the market.
The focus should not be on differentiation of stuff that doesn't matter whether because of some belief that you can out innovate the biggest ecosystem in a utility space or you view it as an on ramp to your own public services. The focus shouldn't be on protecting existing industries (something which concerns me with the EU communication on Cloud). To reiterate for the umpteenth time over umpteen years, the focus should be adapting to this new world and using open source to create a utility market which meets users needs.
Mark Shuttleworth nailed this point many years ago "Innovation and Open Stack : Lessons from HTTP"
On Strategic Play and Open Stack
So when it comes to playing this game, especially because of the powerful effects that ecosystems can create, then the smart strategic play is to build an open source equivalent to EC2/S3/EBS (i.e. all the core AWS features) and create a market place of multiple providers. The goal is to co-opt, sure you can differentiate later when your ecosystem is bigger but at this moment in time co-opt and build a bigger ecosystem through a competitive market.
But how do we know we can match the APIs. Well, the beauty of competitive markets is that they allow for assurance services and exchanges i.e. there is value in the test scripts which ensure that an API is faithful. So, build the test scripts, use those to build your open source IaaS and allow people to create a Moody / Rating agency style business as your market forms.
Whilst CloudStack and Eucalyptus have made steps in this direction, Open Stack (and more likely the RackSpace people involved) seem more reluctant. "Anything but Amazon" seems to be the watch cry and the adoption of EC2/S3/EBS in Open Stack appears to have been something the community forced upon it.
Differentiation on APIs etc is a pointless hangover to the product world. Adoption of APIs and differentiation on Price vs Quality of Service is the way forward. Being an AWS clone and creating a market around such isn't "giving in" but instead it's "competing on what users rather than vendors want".
I would like to see Open Stack succeed, there are many talented people (and friends involved). But I have my doubts because I feel it has wasted an opportunity, it has been very poorly led in my view despite its success in marketing.
The battle in IaaS is going to heat up between AWS and GCE in the next year or so. Open Stack during that time needs to create a large competing market and an equivalent ecosystem which will only be hampered if it doesn't co-opt AWS. Hence it has probably twelve months or so to get the technology rock solid, multiple large scale implementations (by large I mean a minimum $500 million capital spend for each installation), overcome the potential collective prisoner dilemma issue (of everyone differentiating rather than contributing to core) and form the competitive market.
If it fails, then my view is the good ship Open Stack will become the Mary Celeste 2.0 and that will be a terrible waste of effort.
In my view, Open Stack needs a strong benevolent dictator (like a Mark Shuttleworth for Ubuntu). An individual, who is willing to do what's in the interest of the end user and community and ride roughshod over others where necessary. Of course, the essential part is benevolence and its easy to fall foul here. With Open Stack, my advice is they need to focus on engineering quality and build the best open source AWS equivalent there is. Anything else (especially when the words differentiate and innovate are used to describe it) should be unceremoniously dumped of a high cliff for the time being. Along with this should go the every API, every Hypervisor and any "be all to everyone" concepts - focus, focus and more focus on one thing e.g. the best AWS clone. At a push, you could make an argument for AWS and GCE as your native APIs ... but I'd advise against it.
I'm glad we have an Open Stack Foundation, as the chances of that benevolent dictator emerging have grown. I'm hoping someone like Lew Tucker will step up to the plate.
However, where would I place my bet? Currently in the three horse race between Open Stack, Cloud Stack and Eucalyptus then the latter two have been playing a better game in my view though this race is far from over. If forced to choose one, well that's splitting hairs between the likes of Eucalyptus and Cloud Stack but I'd probably bet on Cloud Stack. They have a good focus, they're part of the ASF, they have a well funded backer and they have numerous production deployments at scale. It is however, too early to tell ... the next 12 months are critical.
The issue of standards and in particular open standards is a hotly debated topic. The recent UK Government consultation on open standards was embroiled in much of the politics on the subject with even a media expose of the chair of a consultation meeting as a member of a paid lobbyist group. The rumours and accusations of ballot stuffing of ISO meetings with regards to Microsoft’s OOXML adoption as an open standard are also fairly widespread. The subject matter is also littered with confusing terms from FRAND (fair, reasonable and non discriminatory licensing) being promoted as an “open” standard despite it being IP encumbered by definition.
In general, the principle of standards is about interoperability. In practice, it’s appears more of a battleground for control of a developing market. Standards themselves can also create new barriers to entry into a market due to any onerous cost of implementation. There are also 17 different definitions of what an “open standard” is varying from international bodies to governments. Of these the OSI definition is probably the most respected. Here, open standards are defined as those which have no intentional secrets, not restricted by patents or other technology, no dependency on execution of a license agreement, freely and publicly available and royalty free.
When I talk about evolution of systems (whether activities, practices and data) these are all about meeting specific needs such as providing a web site or carbon dioxide emissions of a country. However standards refer to generic needs that apply across many stages of evolution – the need for things to work together and the need to be able to switch between solutions. These generic needs change in implementation as the underlying system evolves.
For example, in the world of software products then standards can provide interoperability between products and hence the potential to replace one product with another. The standard is normally articulated as a principle of working (i.e. how data is transferred and interpreted) in a document and the expression of this standard (i.e. the code itself) is left up to the product vendor.
This process is often imperfect as one vendor’s interpretation and expression of a principle might not match another’s. However since the switching between solutions is not usually time critical for the end user then some level of imperfection is acceptable i.e. you own the product and if you decide to switch you can migrate at your own pace. Though it should be noted that the process of migration is often a fraught one due to the imperfections.
In the world of utility services, the switching time can be critical and immediate such as the termination of a service. Here imperfections in migration are extremely undesirable and effective switching requires semantic interoperability between the services. In other words, any code and data is understood in the same way between the providers. In practice this can only be achieved if both providers are running the same system or strictly conforming to a reference model (an expression of code) rather than their interpretation of a documented principle.
Switching is an essential part of any competitive market and with utility services then the principle in a standards document is not enough and a reference model (i.e. running code, the expression) is required. If that market is to be unconstrained (i.e. free) then that expression needs to be open itself.
Hence open standards in the product world can simply be open documented principles but in the utility world they require open source reference models (i.e. running code). This simple fact is counter to how standards have been used in the past battles between products and hence the confusion, debates and arguments over the terms are unsurprising.
On Clouds, Open Source and Standards
So, we come to the cloud, which is simply the evolution of a range of IT activities from a product and product rental model to one of commodity and utility services. If we are going to see competitive free markets in this space, then open source reference models are essential. But the value to the end user is the market, not whether this system is more open or not.
An example of this is in the IaaS space. If you ask some of the big users of AWS what they want, they often reply with multiple AWS clones and rarely with another IaaS with another API. To create a marketplace of AWS clones you're going to need to start with an open source system that provides the EC2/S3/EBS APIs with multiple implementations (i.e. providers of such).
I first raised this issue at Web 2.0 in 2006 and by 2007 the whole debate of "open APIs" was kicking off and it has raged ever since. The argument today goes that EC2/S3/EBS are not "open". However APIs are simply principles, they cannot be owned only your expression can be. This has been re-affirmed in several court cases over the last few years.
This debate has already started to switch to "But the process of creating EC2/S3/EBS isn't open" ... well neither is the process for the development of Android and the end user doesn't care. As Benjamin Black once said "Solve user problems or become irrelevant".
The user problem is a competitive free market of multiple utility providers offering the defacto standard (which in case you haven't realised is EC2/S3/EBS) and not a plethora of multiple APIs basically doing the same thing, with super APIs attempting to manage this mess under various cries of being more "Open".
Matt Asay does a good job hitting the point home with his post of "Whose cloud is the open sourciest ... who cares?". Let us be crystal clear here, Matt isn't against open source in the cloud, he (like most of us) understands its essential importance. However, Matt also understands that what needs to be focused on is the user need for competitive markets.
Open source is absolutely essential for creating a competitive free market but the focus should be on solving users need i.e. creating the market.
The focus should not be on differentiation of stuff that doesn't matter whether because of some belief that you can out innovate the biggest ecosystem in a utility space or you view it as an on ramp to your own public services. The focus shouldn't be on protecting existing industries (something which concerns me with the EU communication on Cloud). To reiterate for the umpteenth time over umpteen years, the focus should be adapting to this new world and using open source to create a utility market which meets users needs.
Mark Shuttleworth nailed this point many years ago "Innovation and Open Stack : Lessons from HTTP"
On Strategic Play and Open Stack
So when it comes to playing this game, especially because of the powerful effects that ecosystems can create, then the smart strategic play is to build an open source equivalent to EC2/S3/EBS (i.e. all the core AWS features) and create a market place of multiple providers. The goal is to co-opt, sure you can differentiate later when your ecosystem is bigger but at this moment in time co-opt and build a bigger ecosystem through a competitive market.
But how do we know we can match the APIs. Well, the beauty of competitive markets is that they allow for assurance services and exchanges i.e. there is value in the test scripts which ensure that an API is faithful. So, build the test scripts, use those to build your open source IaaS and allow people to create a Moody / Rating agency style business as your market forms.
Whilst CloudStack and Eucalyptus have made steps in this direction, Open Stack (and more likely the RackSpace people involved) seem more reluctant. "Anything but Amazon" seems to be the watch cry and the adoption of EC2/S3/EBS in Open Stack appears to have been something the community forced upon it.
Differentiation on APIs etc is a pointless hangover to the product world. Adoption of APIs and differentiation on Price vs Quality of Service is the way forward. Being an AWS clone and creating a market around such isn't "giving in" but instead it's "competing on what users rather than vendors want".
I would like to see Open Stack succeed, there are many talented people (and friends involved). But I have my doubts because I feel it has wasted an opportunity, it has been very poorly led in my view despite its success in marketing.
The battle in IaaS is going to heat up between AWS and GCE in the next year or so. Open Stack during that time needs to create a large competing market and an equivalent ecosystem which will only be hampered if it doesn't co-opt AWS. Hence it has probably twelve months or so to get the technology rock solid, multiple large scale implementations (by large I mean a minimum $500 million capital spend for each installation), overcome the potential collective prisoner dilemma issue (of everyone differentiating rather than contributing to core) and form the competitive market.
If it fails, then my view is the good ship Open Stack will become the Mary Celeste 2.0 and that will be a terrible waste of effort.
In my view, Open Stack needs a strong benevolent dictator (like a Mark Shuttleworth for Ubuntu). An individual, who is willing to do what's in the interest of the end user and community and ride roughshod over others where necessary. Of course, the essential part is benevolence and its easy to fall foul here. With Open Stack, my advice is they need to focus on engineering quality and build the best open source AWS equivalent there is. Anything else (especially when the words differentiate and innovate are used to describe it) should be unceremoniously dumped of a high cliff for the time being. Along with this should go the every API, every Hypervisor and any "be all to everyone" concepts - focus, focus and more focus on one thing e.g. the best AWS clone. At a push, you could make an argument for AWS and GCE as your native APIs ... but I'd advise against it.
I'm glad we have an Open Stack Foundation, as the chances of that benevolent dictator emerging have grown. I'm hoping someone like Lew Tucker will step up to the plate.
However, where would I place my bet? Currently in the three horse race between Open Stack, Cloud Stack and Eucalyptus then the latter two have been playing a better game in my view though this race is far from over. If forced to choose one, well that's splitting hairs between the likes of Eucalyptus and Cloud Stack but I'd probably bet on Cloud Stack. They have a good focus, they're part of the ASF, they have a well funded backer and they have numerous production deployments at scale. It is however, too early to tell ... the next 12 months are critical.
