Cloud computing made simple

It has been a truly amazing year since we embarked on our "cloud" journey at Ubuntu, hence I thought I'd review some of the highlights.

We started the journey back in 2008 when Mark Shuttleworth announced our commitment to providing cloud technology to our users. At that time, the cloud world was already in a state of growing confusion, so we adopted an approach of :-

• make the cloud simple.
• focus on one layer of the computing stack (infrastructure) to begin with.
• give our users real technology not promises.
• help drive standardisation (a key requirements of this shift towards a service world) by adopted public defacto standards.
• work with leading partners in this growing industry.
• provide open source systems to avoid lock-in issues.
• actively work to mitigate risks and concerns over cloud by giving our users options.

Hence, in April'09 as part of Ubuntu 9.04 we launched our hybrid cloud strategy.

Our approach was based around the adoption of Amazon EC2 / S3 & EBS as the public defacto standard rather than the creation of some new APIs (there's too many already).

We provided Ubuntu images for use on Amazon EC2 (public cloud) and the technology to build your own private cloud (known as Ubuntu Enterprise Cloud) that matched the same APIs of Amazon. We also added management tools which could cross both public and private domains because of our adoption of a standard API set.

For 9.10 we significantly improved the robustness and ease of setting up a private cloud (Mark built his own several node system in under 25 mins from bare metal). We provided the base for an application store, improved the management capabilities of Landscape and the features of UEC grew extensively. We also launched training, consultancy and support services for the cloud and a JumpStart program to help companies move into the cloud quickly.

During this time we've worked closely with many partners, I'll mention a few (more details can be found on the Ubuntu Cloud site) :-

• Eucalyptus whose open source technology we adopted into the distribution as a core part of Ubuntu Enterprise Cloud.
• Intel's Cloud Builder program to provide best practices on how to create a private cloud using UEC. I'd strongly recommend reading the whitepaper.
• RightScale & CohesiveFT to provide best of breed public management tools alongside our own Landscape system.
• Dell, who will offer a range of pre-built clouds using a series of ‘blueprint’ configurations that have been optimised for different use cases and scale. These will include PowerEdge-C hardware, UEC software and full technical support.

In one year, we've made cloud simple for our users. We've brought our "Linux for Humans" philosophy into the cloud by getting rid of complexity, confusion and myth.

If you want to get into cloud, then we offer :-

• Simple Choices: You can have either private, public or hybrid (i.e. public + private) infrastructure clouds.
• Simple Setup: If you want to build a private cloud, then Ubuntu makes the set-up ridiculously easy. You can be up and running with your own cloud in minutes. Along with with our community documentation covering CD installation and more advanced options, you can also find detailed information on how to build a cloud through Intel's cloud builder program. However, if building a cloud still sounds too daunting then Dell offers pre-built, fully configured and supported private clouds.
• Simple Management: You can use the same tools for both your private and public clouds because we've standardised around a common set of APIs. There's no need to learn one set of systems for private and another for public.
• Simple Bursting: Since we provide common machine images which run on both public and private cloud offerings combined with standardised APIs, then the process of moving infrastructure and combining both private and public clouds is ... simpler.
• Enterprise Help: If you still need help then we offer it, including 24x7 support and a jumpstart program to get your company into the cloud.
• Open source: UEC, the Ubuntu machine images and all the basic tools are open sourced. We're committed to providing open source systems and following through on a genuine open source approach i.e. the system is open source and free and so are all the security patches and version upgrades.

The results of this year have been very encouraging. We recently estimated that there are now over 7,000 private clouds built with UEC, however with 7% of users in our annual Ubuntu User Survey saying that they have built a UEC cloud, the true figure might be very much higher. It was great to hear that almost 70% of users felt Ubuntu was a viable platform for the cloud but there were several surprising statistics including :-

• 54% were using the cloud in some form or another (software, platform or infrastructure). However giving the fuzziness of the term cloud, this can only be seen as a signal of intent to use online services.
• Only 10% had used public cloud providers (such as Amazon) for infrastructure. What was quite remarkable was that given the relatively recent availability of UEC, almost as many people had built private clouds as had used public cloud providers.
• 60% felt that the use of private cloud was more important to their organisation, 25% thought that both private and public was of equal importance whilst only 15% felt that public cloud was the most important.

Whilst this survey was targetted at Ubuntu users, the data we receive from external sources suggest that Ubuntu is becoming the dominant operating system for consumers of the infrastructure cloud space. Even a simple ranking of search terms using Google's Insight around cloud computing show how significant a player Ubuntu is.

Whilst this is great news, what really pleases me is that we're making cloud simple and real for organisations and listening to what they need. We're getting away from the confusion over cloud, the tireless consultant drivel over whether private cloud is cloud computing and the endless pontifications and forums debating vague futures. Instead, we're giving real people, real technology which does exactly what they want.

Over the next year we're going to be tackling issues around creating competitive marketplaces (i.e. more choice for our users), simplfying self-service IT capabilities and orchestration and providing a wide range of open source stacks and platforms to use in the cloud.

We're going to continue to drive down this path of commoditisation by providing common workloads for the cloud (the same as we've been doing for server) and helping businesses to standardise that which is just cost of doing business.

Regardless of any attempts to badge "cloud" as just a more advanced flavour of virtualisation or describe it as "not real yet" by various late vendors, we will be doing our best to bust the various "cloud" myths and push the industry towards competitive marketplaces of computer utilities through defacto standardisation.

Commoditise! Commoditise! Commoditise!

I'm also delighted about our partners successes, with RightScale passing the million server mark, Amazon's continual growth and leadership with the introduction of spot markets, Dell's outstanding move to make cloud mainstream, Intel's push to make cloud easier & Eucalyptus' continued adoption and the appointment of Marten Mickos as CEO.

P.S. if you want to keep track of what's happening with Ubuntu in the cloud, a good place to start is our cloud blog or following our twitter feed, ubuntucloud.

Cloud rant

For the last two posts, I've had a pop at the term "cloud", I'd like to now explain my reasoning in more detail.

First, as always, some background information.

What we know

The transition from a product to a services world in I.T. is very real. It's happening now because of the confluence of concept, suitability, technology and a change it business attitude. Overall it's driven by the process of commoditisation and it is the very ubiquity of specific I.T. activities that makes them potentially suitable for volume operations. I say potentially because volume operations is only viable for activities which are both well defined and ubiquitous.

Fortunately ubiquity has a relationship to certainty (or in other words how well understood, defined and therefore certain an activity is) which is why these activities are suitable for provision on the basis of volume operations through large computer utilities.

Lifecycle

For many years I've talked about lifecycle and the evolution of business activities. Any activity goes through various stages from its first innovation (as per the use of computer resources in the Z3 in 1941) to custom built examples to products which describe the activity. Usually, the activity ends up becoming a ubiquitous and well defined commodity (assuming there are no natural limits or constraints).

During this lifecycle, as the activity becomes more defined in the product stage, service models can often arise (for example the rental model of the managed hosting industry for computing infrastructure or the early subscription like models of electricity provision). As the activity becomes more of a commodity the existence of these service models tends to lead to the rise of utility services (as with electricity provision).

An essential requirement for the growth of the utility model (and the type of volume operations necessary to support it) is that consumers view that what's provided is a commodity. It's little more than a cost of doing business and a standardised version is good enough.

The latter is why a change of attitude is critical in development of the utility service model. If, for example, consumers still view the activity as creating some form of competitive advantage (whether true or not), they are unlikely to adopt a utility model of standard services.

The change in attitude

Over the last decade the attitude of business towards certain I.T. activities has changed dramatically. Recently, a group of 60 odd CIOs & Architects highlighted that many of the I.T. related activities they undertook were commonplace and well defined, particularly across their industry & geography.

Now that doesn't mean they did things the same way, quite the reverse.

Taking just one activity, ERP, then all of these companies agreed that whilst they gain no competitive advantage in ERP, it was an essential cost of doing business. They also agreed that they all had their own customised processes for ERP which they invested heavily in. The shock was their agreement that these different processes provided no differential benefit. It was estimated that for this one activity alone, across this small group of companies, then $600 million p.a. was spent maintaining differences which provided no value.

By reducing customisation through the provision of ERP as standardised services, then each company would significantly benefit in terms of cost savings. Standardisation of processes and removing costs associated with customisation is seen as one of the major benefits of the transition from an "as a Product" to an "as a Service" world.

Let's be clear here, "as a Service" was considered shorthand for "provision of a commodity activity through standardised services via a competitive marketplace of computer utilities". These companies were not looking for an outsourcing arrangement for a highly customised service for their needs, they were looking for a commodity to be treated as a commodity.

The concepts of computer utilities offering elastic and infinite supply on demand, provision of activities through services and commoditisation are all tightly coupled.

The benefits & risks of "cloud"

The benefits of this shift towards service provision via large computer utilities has been discussed extensively for the last 40 years :-

• economies of scale (volume operations)
• focus on core activities (outsourcing to a service provider)
• pay per use (utility charging)
• increased consumer innovation ( componentisation)

However, one critical benefit that gets missed is standardisation itself.

The risks associated with this change (ignoring the disruptive effect on the product industry) can be classified into transitional risks (related to the change in business relationship) and generic outsourcing risks. I've categorised these below for completeness.

Transitional Risks

• Confusion over the new models.
• Trust in the new providers.
• Transparency of relationships.
• Governance of these new models (including auditing, security & management).
• Security of supply

Outsourcing Risks

• Suitability of the activity for service provision.
• Vendor lock-in (& exit costs).
• The availability of second sourcing options.
• Pricing competition.
• Loss of strategic control.

Transitional risks can be mitigated through standard supply chain management techniques. For example with electricity we often combine both public and private sources of provision (a hybrid option). However outsourcing risks require the formation of competitive marketplaces in order to mitigate. Whilst the latter is a genuine concern for companies, even without these marketplaces the benefits of this change are still attractive.

The problem with the term "Cloud"

This change in I.T. is all about standardised service provision of commodity activities through a competitive marketplace of computer utilities. The notion of utility conjures up easily understood and familiar models.

Few would have a problem in understanding how access to a standardised form of electricity through a marketplace has allowed for a huge range of new innovations built upon consuming electricity. Few would have a problem in understanding how the providers themselves have sought new innovative ways of generating electricity. Few would ever consider electricity itself as a form of innovation, to most it comes from a plug and it is critical that it is standardised.

This is a really important point because our companies comprise of value chains that are full of components which are evolving to become commodity and utility like services. This commoditisation enables new higher order systems to be created and new businesses to form e.g. electricity enabled radio, television but also destroys old business. The key here is that whilst commoditisation enables innovation it also destroys the past. The two are different,

The problem with the term "cloud" beyond being fuzzy, is it often used to describe this change in I.T. as something new and innovative. The term helps disguise a fundamental shift towards a world where the bits don't matter and it's all about services. The term allows for all manner of things to be called cloud, many of which have little to do with the standardisation of an activity and its provision through utility services.

You could easily argue the term is misleading as it encourages customisation and distracts the consumer from what should be their focus - standardised services through a competitive marketplace of computer utilities.

Alas, as I said we ALL have to use the term today because of its momentum.

At Ubuntu we focus on commodity provision of activities (common workloads), providing our users with the technology to build a private computer utility (nee "Cloud", as part of a hybrid strategy), the adoption of the defacto standard of EC2 & S3 and we also provide all the technology as open source to encourage the formation of competitive markets.

We use the term "cloud" because it's what customers, analysts and others expect to hear. This of course doesn't stop us from explaining what is really happening and busting the "cloud" myths that exist.

Cloud Camp Frankfurt

A few months ago I provided an introductory talk on cloud computing at Cloud Camp Frankfurt. I was asked to be vendor neutral, so it is light on Ubuntu Enterprise Cloud.

They've put the video of my talk up, so I thought I'd provided some links. Please note, it is split into two parts.

Cloud Computing - Part I

Cloud Computing - Part II

There are more videos on the Cloud Camp Frankfurt site, they're worth watching as the event was a blast.

Lifecycle

For many many years, I've talked about the evolution of activities (from innovation to commodity), how this enables further innovation (componentisation) and why organisations compete in ecosystems (Red Queen Hypothesis). I've also hypothesised an S-Curve of ubiquity vs certainty to describe this evolutionary change, demonstrated techniques to manage such activity life-cycle, shown how Gartner's Hype Cycle can be derived and catalogued the underlying interconnection between Enterprise 2.0, cloud, SOA and web 2.0.

Rather than bore you with the details again, I thought I'd concentrate on a couple of diagrams to explain many of the structural aspects of this.

Figure 1, provides an overview of how activities changes from innovation to commodity, and more importantly how techniques, focus and strategy changes.

Figure 1 - Lifecycle (click on image for higher resolution)

All business activities exists somewhere on this S-Curve and all of them are moving from innovation to commodity. Hence any organisation can be characterised as a network of people interacting with a network of constantly evolving activities, with the organisation itself simply being the intersection between activities and people. You can actually visualise this, though I tend to focus on the network of activities rather than people. By mapping out lines of business against state of evolution, I've found useful tricks for managing activties along with patterns for competing with others.

Learning how to manage activities at their different life-cycle stages and hence knowing how to drive innovation, leverage emerging activities and commoditise that which is cost of doing business is essential for any company. This act is one of balancing the old innovation paradox between survival today (efficiency through co-ordination, coherence & hence order) and survival tomorrow (innovation of new activities, hence deviation, serendipity & disorder).

Since I've often talked about the organisational details of managing life-cycle (the use of pioneers, colonisers and town planners), I thought I'd instead just concentrate on the basic structure. Figure 2 provides the structural elements that are important:-

• Core services : these are core services that the organisation provides, in the I.T. world this is the area where SOA, cloud and other "service" concepts are most relevant.
• Ecosystem : this is the ecosystem that an organisation creates around its core services including workforce, partners, community, channels, customers etc.
• Innovation at "the edge": in general, the larger the ecosystem then the greater the potential for innovation to occur. The concept of innovation at the edge simply refers to expanding the ecosystem as wide as possible.

Figure 2 - Ecosystem (click on image for higher resolution)

In a typical example (e.g. Salesforce & Amazon), the company providing the core services enables and encourages a wider ecosystem to develop around it.

By monitoring new activities (i.e. innovations), and the early adoption of new but similar activities (copying is one of the many signals of success), the company can look to leverage any innovation for the benefit of the wider ecosystem. Methods which can help enable this vary from the provision of an application store to increasing awareness of innovations through the use of Enterprise 2.0 techniques.

By monitoring signals of wider adoption (i.e ubiquity) and early formation of standards (increased definition and certainty of the activity), the company can elect to drive an activity towards commoditisation and provision as a core service. This correspondingly completes the circle and encourages further innovation in the ecosystem through componentisation.

Quite simply put, the structure is simply designed to feed off and accelerate the normal process of evolution of activities (technical or otherwise). This results in a situation where apparent innovation (others are innovating for you), customer focus (leveraging of consumption data to deliver what people want) and efficiency (economies of scale) can all increase with the size of the ecosystem.

I mention this because the centralist approach is to provide all activities at the centre as opposed to "crowdsourcing" both creation and identification of innovation to a wider ecosystem built around a core of common services. Whilst centralisation is not an invalid approach, it is generally highly ineffective when competing against a company which creates a broad ecosystem. This is shown in figure 3.

Figure 3 - Competitive Landscape (click on image for higher resolution)

So with this in mind, I turn to the U.K. Government cloud efforts. It's worth remembering that the U.K. Gov I.T. currently employs over 35,000 people and outsources nearly 65% of its budget (from a total figure of £16bn+). This pretty much makes the internal part of U.K. Gov I.T. equivalent to Google PLUS Amazon whist the outsourcing part is far bigger.

The approach of providing standardised & centralised core services for commodity activities (such as computing infrastructure) is fine. However, as there is no actual competitive utility computing market, an approach of outsourcing to a group of vendors would be unwise (it's worth noting that both Google & Amazon adopted to build in-house). Fortunately, one set of standards - EC2 & S3 - are emerging.

Given this, a sensible strategy would be to adopt these emerging standards, consume conforming external services and where needed build using an open sourced technology (there are several to choose from, Ubuntu Enterprise Cloud would be one). Design for a mix of private / public (hybrid) with a near future view of using multiple public providers.  Develop any needed new elements in-house whilst outsourcing standard components i.e. data centre floor space, power provision etc.

The second area of concern is the Government Application Store. Whilst providing a centralised mechanism of application fulfillment is a sensible approach, the concern should be how those application are developed and provided to the store. Whilst some applications have become commoditised enough to be provided as centrally managed applications or services, it is absolutely essential that the application store should encourage a wider ecosystem (especially at local government level and within communities) for the development of new applications.

A fairly sound approach would be to combine the above with mechanisms of ecosystem monitoring and encouragement for the wider adoption of successful activities. An alternative centralist nightmare, would be where the core (i.e. some form of centralised council) attempts to predict the future whilst defining and developing activities to be adopted (a dis-functional programme management approach). This will lead to the normal flurry of massive development costs, vendor dependency and lock-in.

So why should I care? Well, I live in the U.K. and our Government exists in a wider competitive ecosystem of national governments. The role of U.K. Gov I.T. is to get this stuff right, the same with any other Government.

Platforms and all that jazz ...

Back in early 2006/7 the transition of I.T. activities from a product to service world was often described using three distinct layers - software, framework and hardware. These layers had specific acronyms :-

• SaaS (Software as a Service)
• FaaS (Framework as a Service)
• HaaS (Hardware as a Service).

The terms separated the boundary between what a user was concerned with and what the provider was concerned with.

In the case of HaaS, the provider was concerned with hardware provision (as virtual machines) and the user was concerned with what was built on that hardware (the operating system, any installed framework components such as databases, messaging systems and all code & data).

In the case of FaaS, the provider was concerned with the framework provided (and all the underlying subsystems such as the operating system, the hardware etc) whilst the user was concerned with what they developed in that framework.

I summarised these concepts and the overall stack in a much later blog post during 2007, "The SEP Boundary and more rough thoughts"

Of course, the plethora of 'aaS' terms was foolish especially since the industry had quickly embarked on what can only be described as the 'aaS' wars, a constant renaming of everything. Robert Lefkowitz ( in Jul'07) warned that this was going to lead to a whole lot of aaS. He was right.

Today, those three central layers of the stack have settled on software (though this often yo-yo's to application and back again), platform and infrastructure. However, this seems to have created its own problem.

Platform is being used at all layers of the stack, so we hear of cloud platforms for building IaaS and many other mangled uses of the concepts. The term infrastructure itself has many different meanings. Several SaaS offerings (despite many calling this layer applications, no-one wants to use the acronym AaaS) are described as core infrastructure and of course everything is built with software.

That which is, and still remains very simple, has become fraught with confusion. Life seemed so much simpler back in 2007.

This why, I argued that the CCIF needed to focus on a common taxonomy because we desperately need to talk about the same thing. Now, don't get me wrong, I'm more than aware of the pitfalls of a mechanistic definition of cloud and its inability to impart a wider understanding of the change that is occurring but confusion is a much greater foe.

So, I strongly urge you to adopt the NIST definition for everyday practical use.

The weekly dose ...

Last week, Botchagalupe (irl John Willis) and I finally got around to doing the first of what promises to be a weekly podcast on cloud computing.

After meandering through the dangerous ground of sports fanaticism, we seem to have hit upon a weekly format of :-

• The big question
• What's Hot!
• What's getting the most hype?

We're going to continue it this week but I'd welcome suggestions for the questions we should be discussing. Leave a comment or ping me on twitter.

Is Cloud Computing Green?

The short answer is yes, the long answer is no.

The short answer deals with the better utilisation of computer resources within the data centre, the potential for allocating virtual resources to more energy efficient services and for reducing the massive sprawl of physical resources (and all the energy hence consumed in manufacturing, cooling and distribution.). Overall, cloud computing offers the potential for more energy efficient virtual resources.

The long answer concerns componentisation. The shift of common and well defined I.T. activities from products to standard components provided as online services should lead to a dramatic explosion in innovation.

Standardisation always creates this potential.

If you consider writing an application today, the reason why it's a relatively quick process is because we have relatively standardised and stable components such as frameworks, databases, operating system, cpu, memory etc. Imagine how long it would take to write a new application if you first had to start by designing the CPU. This is componentisation in action, the rate of evolution of system is directly related to the organisation of its subsystems.

Cloud computing is all about providing standard components as services (it's pure volume operations). The problem of course is that we will end up consuming more of these standard components because it's so easy to do so (i.e. in old speak, there is less yak shaving) and it becomes easier to build new and more exciting services on these (standing on the shoulders of giants).

We might end up providing more efficient virtual resources but we will end up consuming vastly more of them.

In the short term, cloud computing will appear to be more green, in the long term it will turn out not to be. However, that's to be expected, our entire history of industrial progress continues to be about the constant creation of ever more complex and ordered systems and the use of stable subsystems simply accelerates this process, whether they be bricks, pipes, resistors, capacitors, databases or whatever.

Whichever way you cut it, our constantly accelerating process of creating greater "perceived" order and the constant reduction of entropy (within these systems and the future higher ordered systems that will be created) ultimately requires one external factor - a greater energy input.

Cloud computing will be no different and our focus will have to be on the source of that energy.

The cloud computing standards war

Over the last couple of years, I've consistently talked about the necessity for standards in the cloud computing space and the oncoming war that this will create. This was not some insightful prediction but simply the re-application of old lessons learned from the many industries which have undergone a transformation to a service world.

That standards war is now in full swing.

The principle arguments behind standards comes from componentisation theory (the acceleration of innovation through the use of standardised subsystems) and the need for marketplaces with portability between providers (solving the lack of second sourcing options and competitive pricing pressures). The two main combatants at the infrastructure layer of the stack are shaping up to be Amazon with the EC2 API and VMware with vCloud API.

Much of the debate seems to be focused about how "open" the standards are, however, there's a big gotcha' in this space. Whilst open standards are necessary for portability and the formation of markets, they are not sufficient. What we really need are standards represented through open source reference models, i.e. running code.

The basic considerations are :-

• A specification can be controlled, influenced and directed more easily than an open source project.
• A specification can easily be exceeded providing mechanisms of lock-in whilst still retaining compliance to a 'standard'.
• A specification needs to be implemented and depending upon the size and complexity of the 'standard' this can create significant adoption barriers to having multiple implementations.
• Open source reference models provide a rapid means of implementing a 'standard' and hence encourage adoption.
• Open source reference models provide a mechanism for testing the compliance of any proprietary re-implementation.
• Adoption and becoming de facto are key to winning this war.

So, in the war of standards whilst the vCloud API has sought approval from the DMTF and formed a consortium of providers, the Amazon EC2 API has widespread usage, a thriving ecosystem and multiple open source implementations (Eucalyptus, Nimbus and Open Nebula).

There appears to be a lot of FUD over the intellectual property rights around APIs and a lot of noise over vCloud adoption. You should expect this to heat up over the next few months because these early battles are all about mindshare and will heavily influence the outcome.

However, whilst VMWare has struck boldly it has exposed a possible achilles heel. The only way of currently implementing vCloud is with VMWare technology, there is no open source reference model. If Reuven is right and Amazon does 'open up' the API, then Amazon have a quick footed route to IETF approval (multiple implementations) and can turn the tables on vCloud by labeling it as a "proprietary" only solution.

Of course VMWare could pre-empt this and go for an open source route or even attempt to co-opt the various open source clouds into adopting their standard. I'd be surprised if they weren't already trying to do this.

This space is going to get very interesting, very quickly.

That Tesla feeling ...

When it comes to the modern electricity industry, Nikola Tesla is undoubtedly one of the most significant figures in its history. His work pioneered the formation of alternating current electrical power systems, including the A/C motor and multi-phase systems for electricity distribution. There are none who would compare in terms of contribution.

By contrast, Thomas Edison was vehemently opposed to the A/C system, even going so far as to publicly electrocuting animals to show its dangers. However, the average person today would likely associate Edison as the "father of modern electricity", in much the same way they might associate him as the inventor of the electric light bulb (as opposed to Joseph Swan).

First, be in no doubt that Edison made enormous contributions to these and many other fields. The question that really needs to be asked is how did Edison become so strongly associated to a field when the contribution of others was equally as large if not greater?

I often nickname this situation as a "tesla moment", an incident in time where the noise generated by others far outweighs the actual contributions made. Nikola Tesla's entire life seems to have been on the wrong side of a prolonged Tesla moment. In my view, he has never really achieved the recognition he deserves.

So why do I mention this? Well, to some extent Canonical has had its own trivial Tesla moment and to be honest this irks me. Canonical, for those of you who don't know, is the company that sponsors and supports Ubuntu - the world's fastest growing linux distribution. In the cloud computing space, Canonical has made some bold moves, including :-

• The first distribution to select KVM as its official hypervisor.
• Launch of officially supported images on Amazon EC2 (2008).
• Integration of Eucalyptus into the distribution (April'09, Ubuntu Enterprise Cloud). We're the first and only distribution to provide their users with a simple way of creating their own clouds that match the Amazon EC2 API (the defacto standard for infrastructure provision in the cloud)
• The introduction of support, training and consultancy services targeted at building private clouds.
• The introduction of officially supported machine images which will run both on a public and private cloud environment across different hypervisors.

We've been working in the background with a number of different partners and we have several announcements aimed for the next release of Ubuntu. Overall, a lot of work has gone into making Ubuntu Server Edition an easy way to get started with cloud computing.

So, you can guess my disappointment that Canonical was not included in the list of 85 vendors shaping the cloud.

Obviously we need to create more noise however we're not going to do this by adding vapour, there's enough already in the cloud world.

Since UEC is freely available, open sourced and doesn't require subscriptions for security updates and patches, there are many people building ubuntu clouds with whom we have no contact. Hence, I'd like to hear from you and how the community is using UEC.

Ping me on twitter.

Cloud definitions ... will it ever end?

At Oscon, I highlighted the problem with many of today's cloud definitions and the attempts to pigeon hole cloud computing as a discrete technology, distribution or billing mechanism.

Such definitions unnecessarily narrow the richness of the field since Cloud computing represents a transition, a shift of many I.T. activities from a product to a service based economy caused by a quartet of concept, suitability, technology and a change in business attitude.

Whilst you can certainly describe some of the mechanics of cloud computing, NIST's latest definition does an excellent job of this, the mechanics do not provide the entire picture. It would be like describing the industrial revolution as :-

"The industrial revolution is a model for enabling convenient, consistent and high speed production of goods that can be rapidly adapted to meet consumer needs. The model promotes availability of goods and is composed of several essential characteristics (use of machinery, higher production volumes, standardised quality, centralised workforce) ..."

A working definition would fail to capture the fundamental transition at play, the history and the potential consequences of this change. It would fail to provide any understanding of what is happening.

An example of this lack of understanding and hence confusion would be the latest debate over what is or is not cloud. This has recently been re-ignited with Amazon's launch of VPC ("virtual private cloud").

In a service economy, a service can be provided by any party and the total of the services provided to an organisation may include both internal and external resources. For example, a company with its own cloud (provided with its own infrastructure, as in the case of an UEC environment) may combine these internal resources with external resources from a provider such as Amazon EC2. This is commonly called the hybrid model.

However, this is a service world and we shouldn't confuse provision with consumption. For example, the company in question is freely at liberty to consume such resources internally (as a private cloud) or to provide those combined resources to others (as a public cloud).

A quick glance at the electricity industry will show you that there exists a wide mix of internal and external resource providers and different models for consumption and provision. I can build my own power generator, top up with the national grid or even sell back to the national grid. The same myriad of different options exists within cloud computing.

Amazon's recent announcement is important because it further enables one of these options. For example, a company could choose to build a private cloud (for internal consumption) which consists of both internal (as in UEC) and external (as in Amazon's VPC) resources. Of course, there are many other ways that this same result can be achieved but none have Amazon's momentum in this field.

The debate over whether it is or is not cloud computing, is simply ... immaterial.

Cloud Computing ... Deja Vu

A new birth always has about it an aura of excitement that be matched by few other spectacles. This is true whether the birth is that of a new being, a new world or a new idea. The excitement arises not so much from the mere fact of birth but rather from the uncertainty and the element of doubt as to the future that always surround a novel event. In this connections, workers in the field computers are now becoming increasingly excited about the birth of a remarkable new method for the distribution and utilization of computer power. This method has been given a variety of names including 'computer utility'.

Regardless of the name, however, the development of this method does open up exciting new prospects for the employment of computers in ways and on a scale that would have seemed pure fantasy only five year ago.

Even now the subject of computer utilities is very much in the public eye, as evidenced by many articles in both the popular and technical press, prognostications by leading industrial and scientific figures and growing signs of interest on the part of governments everywhere.

The word 'utility' in the term 'computer utility' has, of course, the same connotation as it does in other more familiar fields such as in electrical power utilities or telephone utilities and merely denotes a service that is shared among many users, with each user bearing only a small fraction of the total cost of providing that service. In addition to making raw computer power available in a convenient economical form, a computer utility would be concerned with almost any service or function which could in some way be related to the processing, storage, collection and distribution of information.

A computer utility differs fundamentally from the normal computer service bureau in that the services are supplied directly to the user in his home, factory or office with the user paying only for the service that he actually uses.

The computer utility is a general purpose public system that includes features such as :-

1. Essentially simultaneous use of the system by many remote users.
2. Concurrent running of different multiple programs.
3. Availability of at least the same range of facilities and capabilities at the remote stations as the user would expect if he where the sole operator of a private computer.
4. A system of charging based upon a flat service charge and a variable charge based on usage.
5. Capacity for indefinite growth, so that as the customer load increases, the system can expanded without limit by various means.

In addition to the general-purpose public form, there are countless other possible shapes that a computer utility might take. This include private general-purpose systems, public special purpose systems, public and private multi-purpose systems and a whole heirarchy of increasingly complex general-purpose public systems extending all the way to national systems.

As generally envisaged, a computer public utility would be a general purpose public system, simultaneously making available to a multitude of diverse geographically distributed users a wide range of different information processing services and capabilities on an on-line basis.

The public / private division is reflected in our experience with older utilities, communication, gas, electric power etc. In fact, historically, many of our present public utilities began as limited subscriber or private ventures. Even today, despite the fantastic growth of public systems, many organizations continue to operate their own private power plants or internal communication systems.

It is necessary to consider each application of computer utility separately on its merits and balance off in each case the gains and losses resulting from the adoption of the utility concept.

A number of importance considerations tend to improve the cost/effectiveness picture.

1. Reduced solution time for engineering and scientific problems.
2. A capability for an organisation to provide faster service to its customers.
3. Reduce user capital equipment and facility investments.
4. Better utilization of computer resources

Extracts from Douglas Parkhill, The challenge of the computer utility, 1966. (thanks to Tom Wasserman for pointing me in this direction)

Open Clouds

Whilst there are many organisations attempting to define standards for the cloud, my view has always been that these standards will emerge through the marketplace. What is critically important is to the protect the notion of what is and what isn't an open cloud. This is why I actively support the Open Cloud Initiative (OCI) which was founded by Sam Johnston.

The OCI doesn't try to tell you what cloud computing is or isn't, it doesn't even try to tell you what is an open cloud or not. What the OCI does is state, this is our definition of an open cloud (of which there are various forms) and here are the trademarks which you may use to identify your cloud with one of our definitions. He is not saying you must follow our standards or the whims of a committee but instead, he provides a means for end-users to recognise a cloud as being truly open.

The market will decide if Sam's approach will be a success or not but I fully support him in this action.

Why open source clouds are essential ...

I've covered this particular topic over the last four years at various conference sessions around the world. However, given some recent discussions I thought it is worth repeating the story.

"Cloud computing" (today's terminology for an old concept) represents a combination of factors that are accelerating the transition of common IT activities from a product to a service based economy. It's not per se a specific technology but a result of concept, suitability of activities, change in business attitude and available technology (for more information, see my most recent video from OSCON 2009).

The risks associated with this transformation are well known. For example, the risk of doing nothing and the need to remain competitive (see Red Queen Hypothesis part I and part II). This needs to be balanced against standard outsourcing risks (for example: lack of pricing competition & second sourcing options, loss of strategic control, vendor lock-in & suitability of activities for outsourcing) and transitional risks related to this transformation of industry (for example: trust, transparency, governance, security of supply).

These transitional and outsourcing risks create barriers to adoption, however whilst the transitional risks are transitional by nature (i.e. short lived), the outsourcing risks are not. The outsourcing risks can only be solved through portability, easy switching between providers and the formation of a competitive marketplace which in turn depends upon the formation of standards in the cloud computing field. If you want to know more about second sourcing, go spend a few hours with anyone who has experience of manufacturing & supply chain management because this is where the cloud is heading.

Now when it comes to standards in the cloud space, it's important to distinguish that there will be different standards at the various layers of the computing stack (application, platform and infrastructure). People often talk about portability between different layers but each layer is built upon subsystems from the lower layer, you can't just make those magically disappear. You're no more likely to get portability between Azure Platform and EC2 as you are to get portability from a programming language to bare metal (i.e. you need the underlying components).

At each layer of the stack, if you want portability, you're going to need common environments (defined through defacto standards), multiple providers and easy switching between them. For example portability between one Azure environment and another Azure environment.

In the above example, Azure would represent the "standard". However, if a marketplace emerges around a proprietary standard then in effect the entire market hands over a significant element of strategic control to the vendor of that standard.

The use of an open standard (i.e. in this case an open source including APIs and open data formats) is an important mechanism in creating a free marketplace without vendor control. We learnt this lesson from the network wars and the eventual dominance of TCP/IP.

As I've often pointed out, the standard has to be running code for reasons of semantic interoperability. Documented standards (i.e. the principle) are useful but they are not sufficient in the cloud world because of the complexity involved in describing an environment (such as a platform). Even if you could describe such an environment, it would create significant barriers of implementation. 

To achieve the goal of a free market (i.e. free from constraint by one vendor) then you have to solve both the issues of semantic interoperability and freedom from constraint. This means the standard has to be an expression and not principle and the only way to solve the constraint is for the standard to be implemented as an open source reference model (i.e. running code).

This does however lead to a licensing question, if you created an open source reference model for use as a standard, how would you license it? It is important to remember that the intention of a standard is to encourage portability (i.e. limit feature differentiation) but not to limit competition (i.e. to allow differentiation on price vs service quality)

GPLv3 has an important loophole (which I strongly supported and continue to support) known as the "SaaS Loophole" which achieves this goal.

Whilst GPLv3 prevents redistribution of code changes without releasing the modification, it does allow a provider to offer the system as a service with proprietary improvements. GPLv3 encourages competition in the cloud space by allowing providers to operationally "improve" any system and providing it as a service.

In a world where the standard is provided as such an open source reference model (ideally under GPLv3), then you'll also need the creation of an assurance industry to provide end user assurance that providers still match the standard (despite of any competitive modifications for operational improvements). This is how you create a truly competitive marketplace and by encouraging diversity in operations overcome the most dangerous risk of all which is systemic failure in the cloud.

We have already staked the ground with Ubuntu Enterprise Cloud, our intention is to continue to push this and create truly competitive markets in the cloud using the only viable mechanism - open source.  Of course, this is at the infrastructure layer of the computing stack. Our attention will shortly turn towards the platform.

So little time ...

Alas, I've not been blogging recently due to a hectic travel, work and personal schedule. However, for those of you who enjoy my talks, here's the video of my OSCON keynote.

The cloud isn't just vapour.

I was recently asked whether I thought cloud was vapour, whether open sourced cloud systems could solve many of the adoption concerns over cloud computing, whether it was likely that such systems would appear, what standards would they support and which distribution would make the first move? I know I haven't been blogging much recently but I was surprised by the questions.

First, regardless of whatever definition you use to describe cloud computing (a fairly hopeless task in my view), the term merely identifies an underlying shift of I.T. from a product to a service based economy. It's a consequence of :-

1. Certain I.T. activities becoming suitable for service provision through volume operations (i.e. those activities are well defined and ubiquitous
2. The existence of mature enough technology to support this (Popek et al wrote the book on virtualisation back in 1974)
3. A change in business attitude towards I.T. (Carr and others, namely Strassmann, pointed out that much of I.T. is simply seen as a cost of doing business.)
4. The concept of utility computing provision (i.e. the provision of suitable I.T. resources much like other utility providers, as forecast by McCarthy back in the 1960's.)

Take away any of these elements and cloud computing wouldn't represent the upheaval and the disruption that it does today. As for trying to precisely define it, try first coming up with a short and punchy description of industrial revolution without hand-waving and referring to numerous tomes. The problem with cloud computing is that it isn't one thing, it's a transition caused by many factors.

I do believe that open source reference models (or what I used to call open sourced standards) are key to the development of the cloud industry because of the second sourcing concerns of enterprises. I strongly believe that private and hybrid clouds (using both private and public resources) will help develop this industry in the short term. Open source should also dominate this change as it is the only viable route to utility computing marketplaces with competition based upon services rather than lock-in. I've not changed my tune since 2006, I see no reason to change now.

During this time of transition (which is after all what is happening) standards will be incredibly important. Despite all the noise we already have a defacto standard at the infrastructure layer of the computing stack - it's called the Amazon EC2 API.

As for when open source systems will appear that match such emerging standards, the first truly credible system was released almost a year ago. It's called Eucalyptus and it's backed by a commercial company.

As for which distribution would make the first move. Well Ubuntu Server Edition has included Eucalyptus since 9.04. Building a private cloud using open source technology that matches the EC2 API is almost as easy as apt-get install and it's going to get easier. We call this concept Ubuntu Enterprise Cloud and you can find details here.

It's also worth noting that Ubuntu Server Edition is also provided as official images on both UEC and on Amazon EC2, so you can run the same base image in both environments. Matching standards and using uniform images brings us a step closer to portability between environments but most importantly simplifies the process of bursting. It takes us a step further away from the dangers of the cloud net neutrality style argument that I highlighted at E-Tech'07.

It's also essential to build ecosystems around open source cloud computing which is why we work with companies like RightScale and CohesiveFT as well as our own tools like Landscape. Everything we do is around openness and freedom in the cloud computing space. This is not some ideological pursuit but simply a realisation that the future cloud markets will depend upon such openness to form. There is plenty of revenue opportunity without the need to tie people down in an old product mentality.

In short :-

• You can already build clouds with open source technology matching the emerging standard of EC2.
• You can already use single images across both private and public environments.
• The technology is entirely open sourced technology and there exists no lock-in to a proprietary framework or solution.
• It's free.
• It's supported.
• It's already in a distribution, go check out Ubuntu Server Edition.
• You can already use a range of different management tools.

Ubuntu already dominates the linux desktop market, and from the reports I've seen recently we're going great guns on the server market as well. As far as I'm aware, we're the only distribution which provides you with a simple means of creating an open source private cloud and images spanning both private and public environments. Maybe we should get a bit better at shouting about it.

Well, I'm going to be speaking at Velocity and then I have a keynote at OSCON. I was thinking of a tag line for what we've being doing and a friend of mine, Alexis Richardson, chipped in with the following :- "Ubuntu is Cloud for Human Beings"

Perfect.