All in a word.

In my previous post, I provided a more fully fledged version of the lifecycle curve that I use to discuss how activities change. I've spoken about this for many years but I thought I spend a little time focusing on a few nuances.

Today, I'll talk about the *aaS misconception - a pet hate of mine. The figure below shows the evolution of infrastructure through different stages. [The stages are outlined in the previous post]

Figure 1 - Lifecycle (click on image for higher resolution)

I'll note that service bureau's started back in the 1960s and we have a rich history of hosting companies which date well before the current "cloud" phenomenon. This causes a great deal of confusion over who and who isn't providing cloud.

The problem is the use of the *aaS terms such as Infrastructure as a Service. Infrastructure clouds aren't just about Infrastructure as a Service, they're about Infrastructure as a Utility Service.

Much of the confusion has been caused by the great renaming of utility computing to cloud, which is why I'm fairly consistent on the need to return to Parkhill's view of the world (Challenge of the Computer Utility, 1966).

Cloud exists because infrastructure has become ubiquitous and well defined enough to support the volume operations needed for provision of a commodity through utility services. The commodity part of the equation is vital to understanding what is happening and it provides the distinction between a VDC (virtual data centre) and cloud environments.

If you're building an infrastructure cloud (whether public or private) then I'll assume you've got multi-tenancy, APIs for creating instances, utility billing and you are probably using some form of virtualisation. Now, if this is the case then you're part of the way there, so go check out your data centre.

IF :-

• your data centre is full of racks or containers each with volumes of highly commoditised servers
• you've stripped out almost all physical redundancy because frankly it's too expensive and only exists because of legacy architectural principles due to the high MTTR for replacement of equipment
• you're working on the principle of volume operations and provision of standardised "good enough" components with defined sizes of virtual servers
• the environment is heavily automated
• you're working hard to drive even greater standardisation and cost efficiencies
• you don't know where applications are running in your data centre and you don't care.
• you don't care if a single server dies

... then you're treating infrastructure like a commodity and you're running a cloud.

The economies of scale you can make with your cloud will vary according to size, this is something you've come to accept. But when dealing with scale you should be looking at :-

• operating not on the basis of servers but of racks or containers i.e. when enough of a rack is dead you pull it out and replace it with a new one
• your TCO (incl hardware/software/people/power/building ...) for providing a standard virtual server is probably somewhere between $200 - $400 per annum and you're trying to make it less.

Obviously, you might make compromises for reasons of short term educational barriers (i.e. to encourage adoption). Examples include: you might want the ability to know where an application is running or to move an application from one server to another or you might even have a highly resilient section to cope with many legacy systems that have developed with old architectural principles such as Scale-up and N+1. Whilst these are valuable short term measures and there will be many niche markets carved out based upon such capabilities, they incur costs and ultimately aren't needed.

Cost and variability are what you want to drive out of the system ... that's the whole point about a utility. Anyway, rant over until next week.

IT Extremists

The problem with any transition is that inevitably you end up with extremists, cloud computing and IT are no exception. I thought I'd say a few words on the subject.

I'll start with highlighting some points regarding the curve which I use to describe the underlying transition (evolution) behind cloud. I'm not going to simplify the graph quite as much as I normally do but then I'll assume it's not the first time readers have seen this.

Figure 1 - Lifecycle (click on image for higher resolution)

The points I'll highlight are :-

1. IT isn't one thing it's a mass of activities (the blue crosses)
2. All activities are undergoing evolution (commonly known as commoditisation) from innovation to commodity.
3. As activities shift towards more of a commodity, the value is in the service and not the bits. Hence the use open source has naturally advantages particularly in provision of a marketplace of service providers.
4. Commoditisation of an activity not only enables innovation of new activities (creative destruction), it can accelerate the rate of innovation (componentisation) of higher order systems and even accelerate the process of evolution of all activities (increase communication, participation etc).
5. Commoditisation of an activity can result in increased consumption of that activity through price elasticity, long tail of unmet demand, increased agility and co-evolution of new industries. These are the principle causes of Jevons' paradox.
6. As an activity evolves between different stages risks occur including disruption (including previous relationships, political capital & investment), transition (including confusion, governance & trust) and outsourcing risks (including suitability, loss of strategic control and lack of pricing competition.
7. Benefits of the evolution of an activity are standard and include increased efficiencies (including economies of scale, balancing of heterogeneous demand etc), ability of user to focus on core activities, increased rates of agility and tighter linking between expenditure and consumption.
8. Within a competitive ecosystem, adoption of a more evolved model creates pressure for others to adopt (Red Queen Hypothesis).
9. The process of evolution is itself driven by end user and supplier competition.
10. The general properties of an activity changes as it evolves from innovation (i.e. dynamic, deviates, uncertain, source of potential advantage, differential) to more of a commodity (i.e.repeated, standard, defined, operational efficiency, cost of doing business).

The above is a summary of some of the effects, however I'll use this to demonstrate the extremist views that appear in our IT field.

Private vs Public Cloud: in all other industries which have undergone this transition, a hybrid form (i.e. public + private) appeared and then the balance between the two extremes shifted towards more public provision as marketplaces developed. Whilst private provision didn't achieve (in general) the efficiencies of public provision, it can be used to mitigate transitional and outsourcing risks. Cloud computing is no exception, hybrid forms will appear purely for the reasons of balancing benefits vs risks and over time the balance between private and public will shift towards public provision as marketplaces form. Beware ideologists saying cloud will develop as just one or the other, history is not on their side

Commoditisation vs Innovation: the beauty of commoditisation is that it enables and accelerates the rate of innovation of higher order systems. The development of commodity provision of electricity resulted in an explosion of innovation in things which consumed electricity. This process is behind our amazing technological progress over the last two hundred years. Beware those who say commoditisation will stifle innovation, history says the reverse.

IT is becoming a commodity vs IT isn't becoming a commodity: IT isn't one thing, it's a mass of activities. Some of those activities are becoming a commodity and new activities (i.e. innovations) are appearing all the time. Beware those describing the future of IT as though it's one thing.

Open Source vs Proprietary : each technique has a domain in which it has certain advantages. Open source has a peculiarly powerful advantage in accelerating the evolution of an activity towards being a commodity, a domain where open source has natural strengths. The two approaches are not mutually exclusive i.e. both can be used. However, as activities become provided through utility services, the economics of the product world doesn't apply i.e. most of the wealthy service companies in the future will be primarily using open source and happily buying up open source and proprietary groups. This is diametrically opposed to the current product world where proprietary product groups buy up open source companies. Beware the open source vs proprietary viewpoint and the application of old product ideas to the future.

I could go on all night and pick on a mass of subjects including Agile vs Six Sigma, Networked vs Hiearchical, Push vs Pull, Dynamic vs Linear ... but I won't. I'll just say that in general where there exists two opposite extremes, the answer normally involves a bit of both.

A run on your cloud?

When I use a bank, I'm fully aware that the statement I receive is just a set of digits outlining an agreement of how much money I have or owe. In the case of savings, this doesn't mean the bank has my money in a vault somewhere as in all likelihood it's been lent out or used elsewhere. The system works because a certain amount of reserve is kept in order to cover financial transactions and an assumption is made that most of my money will stay put.

Of course, as soon as large numbers of people try to get their money out, it causes a run on the bank and we discover just how little the reserves are. Fortunately, in the UK we have an FSA scheme to guarantee a minimum amount that will be returned.

So, what's this got to do with cloud? Well, cloud (as with banking) works on a utility model, though in the case of banking we get paid on both the amount we consume and provide (i.e interest) and in the cloud world we normally only have the option to consume.

In the case of infrastructure service providers, there are no standard units (i.e. there is no common cloud currency) but instead each provider offers it own range of units. Hence if I rent a thousand computer resource units, those units are defined by that provider as offering a certain amount of storage and CPU for a given level of quality at specified rate (often an hourly fee).

As with any utility there is no guarantee that when I want more, the provider is willing to offer this or has the capacity to do so. This is why the claims of infinite availability are no more than an illusion.

However, hidden in the depths of this is a problem with transparency which could cause a run on your cloud in much the same way that Credit Default Swaps hit many financial institutions as debt exceeded our capacity to service it.

When I rent a compute resource unit from a provider, I'm working on the assumption that what I'm getting is that compute resource unit and not some part of it. For example, if I'm renting on an hourly basis a 1Ghz core with 100Gb storage and 2Gb memory - I'm expecting exactly that.

However, I might not use the whole of this compute resource. This offers the service provider, if they were inclined, an opportunity to sell the excess to another user. In this way, a service provider running on a utility basis could be actively selling 200 of their self defined compute units to customers whilst it only has the capacity to provide for 100 of those units when fully used. This is quaintly given terms like improving utilisation or overbooking or oversubscription but fundamentally it's all about maximising service provider margin.

The problem occurs when everyone tries to use their compute resources fully with an overbooked provider, just like everyone trying to get their money out of a bank. The provider is unable to meet its obligations and partially collapses. The likely effect will be compute units being vastly below their specification or some units which have been sold are thrown off the service to make up for the shortfall (i.e. customers are bumped).

It's worth remembering that a key part of cloud computing is a componentisation effect which is likely to lead to massively increased usage of computer infrastructure in ever more ephemeral infrastructures and as a result our dependency on this commodity provision will increase. It's all worth remembering that black swan events, like bank runs do occur.

If one overbooked provider collapses, then this is likely to create increased strain on other providers as users seek alternative sources of computer resource. Due to such an event and unexpected demand, this might lead to a temporary condition where some providers are not able to hand out additional capacity (i.e. new compute units) - the banking equivalent of closing the doors or localised brown-outs in the electricity industry.

However, people being people will tend to maximise the use of what they already have. Hence, if I'm renting 100 units with one provider who is collapsing, 100 units with another who isn't and a situation where many providers are closing their doors temporarily, then I'll tend to double up the workload where possible on my fully working 100 units (i.e where I believe I have spare capacity).

Unfortunately, I won't be the only one doing this and if that provider has overbooked then it'll collapse to some degree. The net effect is a potential cascade failure.

Now, this failure would not be the result of poor utility planning but instead the overbooking and hence overselling of capacity which does not exist, in much the same way that debt was sold beyond our capacity to service it. The providers have no way of predicting black swan events, nor can they estimate the uncertainty with user consumption (users, however, are more capable of predicting there own likely demands).

There are several solutions to this, however all require clear transparency on the level of overbooking. In the case of Amazon, Werner has made a clear statement that they don't overbook and sell your unused capacity i.e. you get exactly what you paid for.

Rackspace also states that they offer guaranteed and reserved levels of CPU, RAM and Storage with no over subscription (i.e. overbooking).

In this case of VMWare's vCloud Director, then according to James Watters they provide a mechanism for buying a hard reservation from a provider (i.e. a defined unit), with any over commitment being done by the user and under their control.

When it comes to choosing an infrastructure cloud provider, I can only recommend that you first start by asking them what units of compute resource they sell? Then afterwords, ask them whether you actually get that unit or merely a capacity for such depending upon what others are doing? In short, does a compute unit of 1Ghz core with 100Gb storage and 2Gb memory actually mean that or could it mean a lot less?

It's worth knowing exactly what you're getting for your buck.

The bets ...

Every now and then I make bets about the future, a gamble on the actions of individual players (which is, as Hayek would explain, a really bad idea).

I thought I'd write down some of the outstanding bets I have, leaving out the counterparty (unless they're ok with it) & how much it is for (which is always a cup of tea). Naturally these are reckless (I like to gamble) and whilst I don't expect to win, I do expect to get the general direction of travel roughly right. The day I win is the day I haven't pushed it far enough and I'll be disappointed.

[2006] : The Open Source Bet.
By the end of 2015, it will be accepted wisdom that open source provides the only viable mechanism of creating competitive markets in the utility computing world. It will dominate this space and be considered the norm.
[LOST. There's a lot of talk about the importance of open source especially in the cloud market and some good examples such as Cloud Foundry, however the case for creating a competitive market is unproven and open source doesn't dominate the space. Heading in the right direction but NO cigar.]

[2006] : The JS Bet.
By the end of 2015, JavaScript will be seen as a rapidly growing enterprise language for developing entire applications, both client and server side.
[LOST. Despite all the chortling, Javascript has gone on to become a massive language with client and server side components built with this using numerous frameworks. However, I don't think we can say unequivocally that it has become a rapidly growing enterprise language. Heading in the right direction but NO cigar.]

[2007] : The Broker Bet.
By the end of 2013, we will see adverts for computer resource brokers.
[LOST. Alas it looks like it will be end of 2014 / 2015. Heading in the right direction but NO cigar.]

[2008] : The Three Step Bet.
By the end of 2013, Oracle would have bought Sun in order to get into the cloud space, RedHat would have bought Novell and Microsoft will have bought Canonical.
[LOST. Oracle did buy Sun but then RedHat waivered on Novell and without which the triggers for MSFT / Canonical were less likely to happen in the timeframe. NO cigar!]

[2009] : The Escape Button Bet.
By the end of 2014, VMWare will have divided into two companies, one focused on existing virtualisation technology (IaaS Group) and the other focused on PaaS technology. The IaaS group will have been sold off to another company.
[LOST. VMware did build an open source platform play called CloudFoundry and it was this that EMC spun off into a company called Pivotal. However, EMC did not jettison the virtualisation business despite rumours. Heading in the right direction but NO cigar!]

[2010] : The Rackspace Bet.
By Aug 2013, Rackspace will be providing EC2/S3 APIs alongside its own.
[LOST to +James Watters. James argued the team at Rackspace were culturally against such adoption. He was right, I was wrong. Even if by 2016 Rackspace is all over Amazon, NO cigar!]

Added Friday 25th March 2011
Rather than creating new lists, I thought I'd extend this list.

[2011] : The Apple Bet (@GeorgeReese & @mthiele10)
By the end of 2017, Apple will have or will be in the process of filing for protection under chapter 11 of the U.S. bankruptcy code.
[LOST. This was based upon the assumption that Steve Jobs was still in charge and continued a focus on innovative leadership. However, Cook took over and has done a marvellous job in rebalancing the ship. Oh, do I think that AAPL is out of trouble ... not yet ... but Cook has added many years to that company. We're not there yet but this bet is lost. NO cigar!]

Added Friday 30th July 2012
[2012] : The Public Public bet (@lmacvittie)
By the end of 2016 the common view (as in held more often than alternatives) will be that the future is hybrid as in public / public and that private cloud will be seen as niche.
[LOST. Depends upon which circles you walk in but there's enough marketing / vested interests and inertia that there's still a cacophony of hybrid as in public + private. We're close to the tipping point of people waking up not quite there yet. So, NO cigar!]

[2012] : The "Who's the Daddy" bet
The first $1 trillion market share company will be Amazon.
[Hmmm, sadly it looks like I might actually win this one. Obviously didn't push it far enough.]

[2012] The Inertia bet (@jeffsussna)
By mid 2014, a competitive market based around an open source technology which contains at least four IaaS providers (each with >100K servers) that have initiated a price war with AMZN will exist or by 2020 IBM, HP and Dell's server infrastructure business will be in severe decline verging on collapse (unless they have previously exited those businesses).
[Well, the competitive market didn't appear - so we will have to wait until 2020 to see the result]

Added Friday 6th August 2012
[2012] The Modified VMWare Bet (@botchagalupe)
By the end of Feb 2014, VMWare will have :-
1) Acquired Basho or an equivalent company providing distributed storage capabilities to Riak CS
2) Launched an open source IaaS offering.
[LOST to +John Willis. Well VMWare did acquire Virsto and has adopted a closer relationship to OpenStack providing support for it. Heading in the right direction but NO cigar!]

Added Monday 15th October 2012
[2012] The Amazon Two Factor Market Bet (@jeffsussna)
If GCE (Google Compute Engine) launches at scale with success then by end of 2015 Amazon's AWS consoles will have extended to enable you to control your GCE environment.
[LOST to @JeffSussna. AWS didn't launch a console for this but then it is also arguable that Google hasn't launched at either the scale or success that might warrant it. Still, NO cigar. Jeff wins.]

Added 6th May 2013
[2013] AWS Revenue Bet (@geoffarnold)

By end of 2017, AWS revenue will exceed 50% of 2011 worldwide server revenue [approx. $26Bn]

Added Friday 19th July 2013
[2013] The Dead Duck Bet (@cloudbzz, @EdLeafe, @kylemacdonald)

By end of July 2016, either OpenStack will be flourishing through provision of a massive market of AWS clones or the project and its differentiate from Amazon strategy will be widely seen as a dead duck. 

For clarification:-

1. AWS clone does not mean that all AWS services are provided but instead the market is focused solely on providing as many AWS compatible services as possible. The differentiation strategy will have been abandoned or the project will fail.

2. The counter bet is that OpenStack won't be a dead duck nor will there be a massive market of AWS clones built on OpenStack. In this circumstance, the differentiation strategy of OpenStack will be seen as successful.

[LOST to @cloudbzz, @EdLeafe, @kylemacdonald. Well the flourishing market of AWS clones didn't get off the ground and OpenStack pretty much gave up the public space and headed to the private market to compete with VMware, which also adopted OpenStack. Amazon naturally is setting the market ablaze and whilst "OpenStack continues to struggle in its attempts to broaden appeal beyond core customers among service providers and telcos" it has enough supporters along with niches in Telco and projects such as the EC Horizon effort aka Anything but Amazon - that we can't describe it as being widely seen as a dead duck. Certainly in some quarters. We'll just have to wait a few more years. Still, NO cigar!]

Added Monday 9th September 2013

[2013] The Highly Aggressive Smart Phone and Tablet Collapse Bet
(@reverendted, @pwaring )
By the end of 2016, numerous press articles will have proclaimed the future death of smart phones and tablets.  By the end of 2018 both categories will be widely viewed as 'dead markets' and expected to become rapidly niche by end of 2021. NB. The original dates were 2018, 2020 and 2025 respectfully but in this case I took a hyper aggressive stance rather than my normal reckless.

[2013] The It's Alive! Bet (@crankypotato)

By the end of 2023, research labs will have an entirely biological clock including display with no mechanical or electronic components.

Added Saturday 13th September 2014

[2014]  The "I'm not allowed to drive here anymore" Bet (@h_ingo and @lhochstein)
By end of 2030, at least one city with a population in excess of 1 million will have outlawed people driving cars.

[2014]  The "Silicon Valley isn't what it used to be!" Bet (@quentynblog)
By end of 2030, Silicon Valley will not be commonly ranked in the top 5 most innovative places in the world.

[2014] The Amazon doesn't lose Bet (@littleidea)
By end of 2025, then neither MSFT nor Google will have wiped out AMZN which will remain a dominant player in cloud infrastructure. 'Wiped out' will mean less than 15% of market share by use. 'Dominant' will mean in the top 2 of public infrastructure providers by use.

For clarification
* 15% then I buy the tea
* Still in the top 2, then @littleidea buys the tea
* 15% but out of top 2 then we just hang out drinking tea

[2014] The Disruption Bet (@crankypotato)

By end of 2030, the period 2025 to 2030 will be commonly considered (by the general press, population and business press) as far more 'disruptive' than the period 2010 to 2020.

[2014] The Big Data's gone horribly wrong Bet [@thinkinnovation]
By end of 2023, Big Data will have experienced its first casualties with more than Five Big Data (product) vendors having gone bust or sold in a fire sale. For clarification, the vendor must be a well known name to experts in the field in 2014.

[2014] The IoT didn't save us Bet [@ZakKissel]

By end of 2030, many of today's large Tech company leaders in IoT will be reported (in the common tech press) as facing severe disruption by new entrants. For clarification. Many will be four or more company names associated as tech leaders in the field in 2014. Large will mean greater than 5,000 employees.

[2015] The Warren Buffet Bet[@jstodgill]
By the end of 2030, Warren Buffet will have made a small fortune from printed electronics. Key metrics will be that Warren Buffet will have a) invested in printed electronics b) made a 10x return.

Added 3rd July 2015
[2015] The "I blew the company but still got paid millions, it was the staff wot dunnit" executive sweepstake.
By the end of 2025, how many of these companies will survive as stand-alone entities? For clarification, being acquired does not count as surviving nor does merging with another. The candidate companies are - Cisco, IBM, HP, Oracle, Microsoft, SAP, Dell, NetApp, VMware, Amazon and RedHat

0 survive - @frabcus

1 survive - @rorti33

2 survive - @zeruch

3 survive - me (@swardley)

4 survive - @sddc_steve 

5 survive - @saulcozens  

6 survive - @WorkingHardInIT

7 survive - @rbramley (would prefer 6 but that was taken).

8 survive - @codebeard (for a laugh, though he'd rather have a lower number).

9 survive - @yoz (though he'd rather have 7)

10 or 11 survive  - alas, no-one was willing to take this. Rather telling.

To be honest there was much nashing of teeth that the lower numbers had gone and so the high numbers bets are more gestures of sporting behaviour. The usual bet, I'll buy a cup of tea for the winner next time I see them.

On a general note, everyone expects some - if not a a significant number of these companies to be gone by 2025. So, if you're into collectables ... do remember this. There might be some useful freebies at events that you can hold onto for 40 years or so as reminders of past greatness.

Added 25th August 2015

[2015] The "We succeeded by changing the definition" bet. [@WorkingHardInIT] 
By end 2020 Gartner will state that :-
1) hybrid means public plus public cloud consumption i.e. not just public plus own hosted (i.e. private) environments.
2) bimodal means three (or more) groups, not two.

[2015] The I had to up the risk factor (i.e. drop from 10%) until someone accepted - "OpenStack is sooooo niche" bet. [@Vecchi_Paolo]
By the end of 2020, the total revenue from product, direct licensing and service provider sales (excluding general hardware) of the entire OpenStack ecosystem will be less than 1% of the revenue generated by AWS.

To be frank, I expect that OpenStack will make a reasonable sum of money in the network equipment vendors space ... however, it's only a cup of tea. Still can't believe I had to drop from 10% to 1% of AWS revenue to get any takers. Not exactly a great deal of confidence out there.

Added 8th December 2016

[2016] The somewhat extreme mode "Hey, just what happened to Uber?" bet [@CloudOpinion, @GeorgeReese]
By the end of 2024, Didi Chuxing will dominate the US self driving taxi market.
Had to reduce from 2025 before anyone would happily take it.

[2016] The extreme mode "your entire company runs on Lambda?" bet [@davidajbagley, @jmwiersma, @LarryLarmeu]
By end of 2021, a $5Bn revenue company will announce it is running entirely on AWS lambda.
Had to reduce the date from 2024 all the way to 2021 before anyone would take it.

[2016] The somewhat extreme "What's an FPGA?" bet [@bmkatz]
By the end of 2023, a modular FPGA based device will dominate the mobile market.
Original had a target of 2025 but hey ... extreme is extreme.

Arguably, the best cloud conference in the world?

For those of you who missed the OSCON Cloud Summit, I've put together a list of the videos and speakers. Obviously this doesn't recreate the event, which was an absolute blast, but at least it'll give you a flavour of what was missed.

Welcome to Cloud Summit [Video 14:28]
Very light introduction into cloud computing with an introduction to the speakers and the conference itself. This section is only really relevant for laying out the conference, so can easily be skipped.
With John Willis (@botchagalupe) of opscode and myself (@swardley) of Leading Edge Forum.

Scene Setting
In these opening sessions we looked at some of the practical issues that cloud creates.

Is the Enterprise Ready for the Cloud? [Video 16:39]
This session examines the challenges that face enterprises in adopting cloud computing. Is it just a technology problem or are there management considerations? Are enterprises adopted cloud, is the cloud ready for them and are they ready for it?
With Mark Masterson (@mastermark) of CSC.

Security, Identity – Back to the Drawing Board? [Video 25:12]
Is much of the cloud security debate simply FUD or are there some real consequences of this change?
With Subra Kumaraswamy(@subrak) of Ebay.

Cloudy Operations [Video 22:10]
In the cloud world new paradigms and memes are appearing :- the rise of the “DevOps”, “Infrastructure == Code” and “Design for Failure”. Given that cloud is fundamentally about volume operations of a commoditized activity, operations become a key battleground for competitive efficiency. Automation and orchestration appear key areas for the future development of the cloud. We review current thinking and who is leading this change.
With John Willis (@botchagalupe) of opscode.

The Cloud Myths, Schemes and Dirty Little Secrets [Video 17:38]
The cloud is surrounded by many claims but how many of these stand up to scrutiny. How many are based on fact or are simply wishful thinking? Is cloud computing green, will it save you money, will it lead to faster rates of innovation? We explore this subject and look at the dirty little secrets that no-one wants to tell you.
With Patrick Kerpan (@pjktech) of CohesiveFT.

Curing Addiction is Easier [Video 18:41]
Since Douglas Parkhill first introduced us to the idea of competitive markets of compute utilities back in the 1960s, the question has always been when would this occur? However, is a competitive marketplace in the interests of everyone and do providers want easy switching? We examine the issue of standards and portability in the cloud.
With Stephen O’Grady (@sogrady) of Redmonk.

Future Setting
In this section we heard from leading visionaries on the trends they see occurring in the cloud and the connection and relationships to other changes in our industry.

The Future of the Cloud [Video 29:00]
Cloud seems to be happening now but where is it going and where are we heading?
With J.P. Rangaswami (@jobsworth) of BT.

Cloud, E2.0 – Joining the Dots [Video 30:04]
Is cloud just an isolated phenomenon, or is it connected to many of the other changes in our industries.
With Dion Hinchcliffe (@dhinchcliffe) of Dachis.

The Questions
The next section was a Trial by Jury where we examined some of the key questions around cloud and open source.

What We Need are Standards in the Cloud [Video 45:17]
We put this question to the test, with prosecution Benjamin Black (@b6n) of FastIP, defence Sam Johnston (@samj) of Google and trial by a Jury of John Willis, Mark Masterson, Patrick Kerpan & Stephen O’Grady

Are Open APIs Enough to Prevent Lock-in? [Video 43:21]
We put this question to the test, with prosecution James Duncan (@jamesaduncan) of Joyent, defence George Reese (@georgereese) of Enstratus and trial by a Jury of John Willis, Mark Masterson, Patrick Kerpan & Stephen O’Grady

The Debates
Following the introductory sessions, the conference focused on two major debates. The first of these covered the “cloud computing and open source question”. To introduce the subject and the panelists, there were a number of short talks before the panel debates the impact of open source to cloud and vice versa.

The Journey So Far [Video 10:59]
An overview of how “cloud” has changed in the last five years.
With James Urquhart (@jamesurquhart) of CISCO.

Cloud and Open Source – A Natural Fit or Mortal Enemies? [Video 8:44]
Does open source matter in the cloud? Are they complimentary or antagonistic?
With Marten Mickos (@martenmickos) of Eucalyptus.

Cloudy Futures? The Role of Open Source in Creating Competitive Markets [Video 8:43]
How will open source help create competitive markets? Do “bits” have value in the future and will there be a place for proprietary technology?
With Rick Clark (@dendrobates) of OpenStack.

The Future of Open Source [Video 9:34]
What will cloud mean to open source development and to linux distributions. Will anyone care about the distro anymore?
With Neil Levine (@neilwlevine) of Canonical.

The Debate – Open Source and the Cloud
 [Video 36:24]
Our panel of experts examined the relationship between open source and cloud computing.
With Rick Clark, Neil Levine, Marten Mickos & James Urquhart

The Future Panel followed the same format with first an introduction to the experts who will debate where cloud is going to take us.

The Government and Cloud [Video 10:27]
The role of cloud computing in government IT – an introduction to the large G-Cloud and App Store project under way in the UK; what the UK public sector hopes to gain from a cloud approach, an overview of the proposed technical architecture, and how to deliver the benefits of cloud while still meeting government’s stringent security requirements.
With Kate Craig-Wood (@memset_kate) of Memset.

Infoware + 10 Years [Video 10:38]
Ten years after Tim created the term infoware, how have things turned out and what is the cloud’s role in this?
With Tim O'Reilly (@timoreilly) of O'Reilly Media.

The Debate – A Cloudy Future or Can We See Trends? [Video 50:12]
The panel of experts examine what’s next for cloud computing, what trends can they forsee.
With Kate Craig-Wood, Dion Hinchcliffe, Tim O’Reilly & JP Rangaswami

So, why "arguably the best cloud conference in the world?"

As a general conference on cloud, then the standard and quality of the speakers was outstanding. The speakers made the conference, they gave their time freely and were selected from a wide group of opinion leaders in this space. There was no vendor pitches, no paid for conference speaking slots and hence the discussion was frank and open. The audience themselves responded marvelously with a range of demanding questions.

It is almost impossible to pick a best talk from the conference because they were all great talks. There are real gems of insight to be found in each and every one and each could easily be the keynote for most conferences. In my opinion, if there is a TED of cloud, then this was it.

Overall, the blend of speakers and audience made it the best cloud conference that I've ever attended (and I've been to 50+). This also made my job as a moderator simple.

I'm very grateful to have been part of this and so my thanks goes to the speakers, the audience, the A/V crew who made life so easy and also Edd Dumbill (@edd), Allison Randal (@allisonrandal), Gina Blaber (@ginablaber) and Shirley Bailes (@shirleybailes) for making it happen.

Finally, huge thanks to Edd and Allison for letting me give a version of my Situation Normal, Everything Must Change talk covering cloud, innovation, commoditisation and my work at LEF.

Islands in the sky

I'm often asked how will the cloud develop to which I'll answer -"imperfectly, very imperfectly".

I was reminded of this through a long discussion with Benjamin Black, hence I thought I'd write something to explain my general thoughts on the problem. First, let me apologise as this will be a long post. Second, we need to start by recaping some basic concepts about risks. The barriers to adoption in cloud cover three basic forms of risk :-

Disruption Risks : Change to existing business relationships combined with issues around political capital and previous enterprise investment. It's often difficult to let go of that which we have previously invested in.

Transitional Risks: These risks are related to the shift from a world of products to a world of services and they include confusion over the models, trust in the service providers, governance of this service world, transparency from the providers and security of supply. Many of the transitional risks can be mitigated with a hybrid (private + public) cloud approach, a standard supply chain management technique. This approach has been used in many industries which have undergone a similar change, for example in the early decades of power generation it was common to combine public generation with private generators. Even today most data centres mix a variety of public suppliers with backup generators and UPS systems. Fortunately, these transitional risks are relatively short lived.

Outsourcing Risks: These cover lack of pricing competition between the new providers , lack of second sourcing options between providers, loss of strategic control to a specific technology vendor, lock-in and unsuitability of the activity for such service provision (i.e. it's not ubiquitous or well defined enough for such volume operations based service provision). The outsourcing risks can be reduced through the formation of a competitive marketplace of providers with easy switching between them and ideally the option to in-house service provision. The outsourcing risks are long term.

For a competitive market to form, you need easy switching which means portability. The basic ingredients of portability include a choice of providers, access to your code and data from any provider and semantic interoperability between providers i.e. both the origin and destination providers need to understand your code and data in the same way. There is limited value in having access to your code and data if no other provider understands it and operates to provide the same functionality e.g. getting access to your data in salesforce is great but what do you do with it?

In such circumstances, there does exist a weaker form of syntactic interoperability, which means both providers can exchange data but the end result may not function in the same way and your data may not retain its original meaning. Often, this is where we see translation systems to convert from one system to another with the usual abundance of translation and semantic errors.

The ideal situation is therefore semantic interoperability, which generally means a common reference model (i.e. running code) which providers either operate or conform to. Unfortunately, common reference models come with their own risks.

Let us suppose you have a marketplace of providers offering some level of service at a specific level of the computing stack (SPI Model) and these providers operate to a common reference model. The model provides APIs and open data formats, giving you access to your code and data. You therefore have a choice in providers, access to your data and semantic interoperability between them. You have portability. BUT, if that common reference model is owned by a vendor (i.e. it's proprietary code) then that market is not free of constrant but instead controlled by the vendor. All the providers & consumers in that marketplace hand over a significant chunk of strategic control and technology direction to the vendor, who is also able to exert a tax on the market through license fees.

To reduce this loss of strategic control and provide a free market (as in free of constraints), then that common reference model must not be controlled by one party. It has to be open sourced. In such an environment, competition is all about operational efficiency and price vs QoS rather than bits. This makes intuitive sense for a service world, which is why I'm pleased openstack is following that route and I hope it will become the heart of a market of AWS clones. Obviously, you'll need different common reference models at different layers of the computing stack. Whilst only one is probably needed for infrastructure, you will need as many as there are competitive application marketplaces (CRM, ERP etc) in the software later of the SPI model.

Before anyone cries the old lie of standardisation hampers innovation, it's worth remembering that utility service provision (which is what cloud is really about) requires volume operations which in turn requires a ubiquitous and well defined activity. Whilst the common reference models certainly won't be perfect in the beginning, they don't need to be, they only have to create "good enough" components (such as a defined virtual machine). They will improve and evolve over time but the real focus of innovation won't be on how good these "good enough" components are but instead what is built with them. This concept, known as componentisation, is prevalent throughout our industrial history and shows one consistent theme - standardisation accelerates innovation.

So everything looks rosy … we'll have the economics benefits of cloud (economies of scale, increased agility, ability to focus on what matters), competitive marketplace based around multiple providers competing on price vs QoS, the options to use providers or install ourselves or to mitigate risks with a hybrid option, "open" API & data formats giving us access to our code and data, open sourced common reference models providing semantic interoperability, "good enough" components for ubiquitous and well defined activities which will cause an acceleration of innovation of new activities based upon these components … and so on.

Think again.

In all likelihood, we're going to end up with islands in the cloud, marketplaces built around specific ways of implementing a ubiquitous and well defined activity. Don't think of "good enough" components but instead a range of different "good enough" components all doing roughly the same thing. Nuts? It is.

Hence, in the infrastructure layer you're likely to see islands develop around :-

• EC2/S3 (e.g. core of AWS) including the open source implementations such as Open Stack, Eucalyptus and Open Nebula.
• vCloud principally provided through VMWare technology.
• a Microsoft infrastructure based environment.
• any Openstack APIs, particularly if Rackspace implements this.

All of these will be providing their own versions of "good enough" units of virtual infrastructure. Within those islands you'll head towards multiple service providers or installations, a competitive marketplace with switching between installation and semantic interoperability based upon a common reference model. The open source projects such as OpenStack are likely to form assurance industries (think moody's rating agencies, compliance bodies) to ensure portability between providers by comparison to the common reference model whereas the proprietary technologies are likely to develop certification bodies (e.g. VMWare Express).

Between islands there will be only syntactic interoperability (with exceptions such as OpenStack which will try to span multiple Islands), which will mean that you'll require translation of systems from one island to another. Whilst management tools will develop (and already have started) to cover multiple islands and translation between them, this process is imperfect and a constant exercise in chasing different APIs and creating a lowest common denominator (as per libcloud). Of course, I wouldn't be surprised if the libcloud folk were hoping that as a community develops around them, then the providers will offer libcloud as a native API. Such command & conquer strategies rarely succeed.

Given this complexity and since there will be multiple service providers within an island, it's likely that consumers will tend to stick within one island. If we're lucky, some of these Islands might die off before the problem becomes too bad.

Of course, these base components could effect the development of higher order layers of the computing stack and you are likely to see increasing divergence between these islands as you move up the stack. Hence, the platform space on the vCloud island will differ from the platform space on the EC2 / S3 island. We will see various efforts to provide common platforms across both, but each will tend towards the lowest common denominator between the islands and never fully exploit the potential of any. Such an approach will generally fail compared to platforms dedicated to that island, especially if each island consists of multiple providers hence overcoming those general outsourcing risks (lack of second sourcing options etc). Maybe we'll be lucky.

So, the future looks like multiple cloud islands, each consisting of many service providers complying to the standard of that island - either vCloud, EC2/S3 or whatever. Increasing divergence in higher order systems (platforms, applications) between the islands and whilst easy switching between providers on an island is straightforward, shifting between islands requires translation. This is not dissimilar to the linux vs windows worlds with applications and platforms tailored to each. The old style of division will just continue with a new set of dividing lines in the cloud. Is that a problem?

Yes, it's huge if you're a customer.

Whilst cloud provides more efficient resources, consumption will go through the roof due to effects such as componentisation, long tail of unmet business demand, co-evolution and increased innovation (Jevons' paradox). Invariably one of the islands will become more price efficient i.e. there is no tax to a technology vendor who collects their annual license and upgrade fee through a drip feed process. It's this increased dependency combined with price variance which will result in operational inefficiencies for one competitor when compared to another who has chosen the more efficient island. The problem for the inefficient competitor will be the translation costs of moving wholesale from one island to another. This is likely to make today's translations look trivial and in all probability will be prohibitive. The inefficient competitor will be forced therefore to compete on a continual disadvantage or attempt to drive the technology vendor to reduce their taxation on the market.

The choices being made today (many are choosing islands based upon existing investment and political choices) will have significant long term impacts and my come to haunt many companies.

It's for these reasons, that I've recommended to anyone getting involved in cloud to look for :-

1. aggressively commoditised environments with a strong public ecosystem.
2. signals that multiple providers will exist in the space.
3. signals that providers in the space are focused on services and not bits.
4. an open source reference implementation which provides a fully functioning and operating environment.

In my simple world, VMWare is over-engineered and focuses on resilient virtual machines rather than commodity provision. It's ideal for a virtual data centre but we're talking about computing utilities and it also suffers from being a proprietary stack. Many of the other providers offer "open" APIs but as a point of interest APIs can always be reverse engineered for interoperability reasons and hence there is no such thing as "closed" API.

The strongest and most viable island currently resides around EC2 / S3 with the various open source implementations (such as UEC), especially since the introduction of Rackspace & Nasa's service focused openstack effort.

I don't happen to agree with Simon Crosby that VMWare's latest cloud effort Redwood == Deadwood. I agree with his reasoning for why it should be, I agree that they're on shaky grounds in the longer term but unfortunately, I think many companies will go down the Redwood route for reasons of political capital and previous investment. IMHO I'm pretty sure they'll eventually regret that decision.

If you want my recommendation, then at the infrastructure layer get involved with open stack. At the platform layer, we're going to need the same sort of approach. I have high hopes for SSJS (having been part of Zimki all those years back), so something like Joyent's Smart platform would be in the right direction.

---  Added 19th August 2013

Gosh, this is depressing. 

Three years and 15 days later Ben Kepes (a decent chap) writes a post on how we're coming to terms with what are basically "islands in the clouds".

OpenStack followed a differentiation road (which James Duncan and I raised as a highly dubious play to the Rackspace Execs at the "OpenStack" party in July at OSCON 2010). They didn't listen and we didn't get the market of AWS clones. In all probability if AWS compatibility had been the focus back in 2010 then the entire market around OpenStack could have possibly been much larger than AWS by now. But, we will never know and today, OpenStack looks like it has almost given up the public race and is heading for a niche private role.

In his article, Ben states that companies never wanted "cloud bursting" - a term which seems to be a mix of 'live' migration (a highly dubious and somewhat fanciful goal to aim for which is more easily managed by other means) combined with the ability to expand a system into multiple environments.

Dropping the 'live' term, then both can be achieved easily enough with deployment and configuration management tools. One of the reasons why I became a big fan of Chef in '08/'09 (and not just because of my friend Jesse Robbins). This sort of approach is simple if you have multiple providers demonstrating semantic interoperability (i.e. providing the same API and the same behaviour) as your cost of re-tooling and management is small. It becomes unnecessarily more complex with more Islands.

Anyway, that aside the one comment I'll make on Ben's post is the goal was never "cloud bursting" but instead second sourcing options and balancing of buyer / supplier relationship. Other than that, a good but depressing post.

Private vs Public clouds

I thought this argument has been settled a long time ago, seems not. So, once more dear friends I will put on my best impression of a stuck record. First what is the difference between a public and a private cloud?

• A public cloud (the clue is in the name) is open to the public.
• A private cloud (the clue is in the name) is private to some set of people.

Naturally, there are shades of grey. For example, the set of people for which a private cloud is private might be one person, a department, a company, a community, a nation or some sort of collection of the above. It is common to use a variety of notations (community, government etc) to distinguish these constraints on use i.e. what subset of people are allowed to use it.

There is another side to this which is your relationship to the provider. It is either :-

• external to you and therefore controlled, operated and run by another party.
• internal to you which means it is controlled, operated and run by yourself.

Now, once again there are shades of grey because it is perfectly possible for a community of companies to build a shared cloud environment. Examples of the notation include :-

• AWS offers a public cloud which is external to everyone but Amazon.
• Eucalyptus offers technology for a company to build a private cloud which is internal to that company.

You could write a list with examples of each but there is little point as no-one uses this notation. Instead in common parlance we tend to use the term public cloud with a single counterpoint of private cloud to mean a cloud where an organisation makes up the subset of private users and the cloud is provided internally to that organisation. Now we have our bearings on the terms, this leaves a question ...

Why use a private cloud?

A private cloud (using the common meaning) is one that you control and operate. It hence overcomes - or at least creates the illusion of overcoming - many transitional risks such as governance, trust & security of supply. However, it does so at the potential loss of the economies of scale found in public clouds combined with additional costs such as planning, administration and management.

The choice over whether to use one type of cloud or another is always one of benefits vs risks (whether disruption, transition or outsourcing risks). A hybrid cloud strategy simply refers to using a combination of both public and private clouds to maximise benefits for a given appetite of risk.

Naturally, the actual risk can change with a variety of events. For example, the formation of competitive cloud marketplaces with easy switching between multiple providers reduces outsourcing risks (e.g. lack of pricing competition, loss of strategic control, lack of second sourcing options).

For a consumer of cloud services, the ideal scenario is multiple providers of the same service, the option to implement your own environment and no loss of strategic control or dependency to a single vendor. For this to scenario happen, the technology must be open sourced and hence the technology owners must first realise that in this cloud world value isn't generated through bits of software but instead through services.

In the same way that it took a book company to disrupt the hosting world by offering commoditised infrastructure services, a hosting company is now trying to do the same to the world of technology vendors through open source. This is just the start and whilst openstack is currently focused on the infrastructure layer, expect it to move up the computing stack in short order.

There are four companies who in my mind exemplify this whole commodity approach - Rackspace, Amazon, Google and Canonical. I expect they will become titans in this space.

-- 5th January 2017

In 2017, there still exists a surprising view that private cloud is more than transitional. Whilst Amazon, Google and Canonical have emerged as major players (in the case of Amazon, a Titan) the story of OpenStack and Rackspace was less rosy. A mix of poor strategic gameplay led to OpenStack becoming firmly entrenched in the private cloud space and though it move up the stack a bit, it never moved into the platform space. That charge has been left to Cloud Foundry which is now facing off against Amazon's effort - Lambda. Rackspace lost the public cloud battle but it has re-invented itself around Amazon. 

OSCON 2010

I thoroughly enjoyed the OSCON cloud summit and the talk that I gave at OSCON - the audiences were fantastic and the organisation was superb (huge thanks to Edd, Allison and the O'Reilly crew for making this happen).

I'm really proud to have played my small part in this event as the MC for day, along with John Willis.

I haven't yet talked a great deal on my research, but the keynote at OSCON gives a taste of it - so I thought I'd link to it here. Those who know me, also know that this had been a hobby horse of mine over the last decade. It's finally good to spend some focused time on it though of course these ideas are far from new.

A couple of final notes :-

• Utility services are just a domain within the commodity phase of an activity's evolution. There are constraints which will prevent a commodity being provided through services. I sometimes plot on the graph a wider "services" stage, however for the sake of simplicity I've left this out.
• The stages of lifecycle are approximate only i.e. this is where products appear, this is where utility services generally appear etc.
• Multiple activities can be bundled into a single product. For example the iPhone is a combination of different activities from personal communication to digital recorder to web surfing to time keeper to ... the list is quite long. These activities are all evolving and being implemented by others, which forces Apple to focus on two areas :- the bundling of new innovative activities into the iPhone and application innovation through the App Store. The former is expensive and risky. The later requires development of a strong ecosystem, ideally with users being allowed to create and distribute their own applications. The manner in which Apple manages this is less than ideal and they now face severe disruption from Android. As there is also little exploitation of the wider manufacturers' ecosystem, Apple has cornered itself into creating highly costly & risky innovations with weak leveraging. IMHO, they are in trouble and this should become painfully clear in the next five years unless they change.
• The ILC model is generally applicable. I picked examples from cloud providers but equally I could have discussed Canonical with Ubuntu. Canonical ruthlessly commoditises activities to provide a stable core and I'd strongly argue that Rackspace & Canonical point to the future direction of IT.
• Open source is the natural end state for any activity described by software which is ubiquitous and well defined. This doesn't mean that open source can't be used earlier, of course it can and there are numerous tactical advantages of doing so, along with benefits such as increased collaboration. However, what I am saying is that by the time an activity has reached the commodity phase then only open source makes sense. Those who have been questioning whether "cloud is the death of open source" have a poor understanding as to what is actually happening.
• Open core is in general a tactical anomaly. On the one hand, if successful, it will cause widespread distribution (driving an activity towards more of a commodity) and yet it attempts to generate revenue through proprietary elements which is against the natural state that open core is forcing activities towards. A number of companies have used this approach successfully and have even been bought for huge sums by large companies. However, it still remains a tactical anomaly which attempts to achieve both the benefits of open and closed by being both.
• The S-Curves I use are not time based. If you follow the evolution of an activity through specific phases of its lifecycle and plot adoption against time, you will derive a set of non-uniform S-Curves for Roger's diffusion of innovation. It's important to realise that the accelerators I mentioned (open source, participation, network effects) along with others I didn't mention (communication mechanisms, co-evolution etc) alter the speed at which an activity evolves. Whilst, this doesn't impact the S-Curves I use, it does compact Roger's curves of more recent innovations when compared to earlier diffusions.
• The speed at which an activity moves across the profile graph (i.e. through its lifecycle) depends upon the activity.
• None of these ideas are new. The nearest to new is company profile which I've been refining in the last year from earlier work (between '04-'07) and this refinement is simply a formalisation of already existing concepts. If you watched the video and thought, "that's new", then my only advice is be concerned.
• On the question of science, the models presented (S-Curve, Profile) are part of a general hypothesis on the evolution of business activities. Whilst data exists, there is neither the volume of evidence nor independent observation to validate beyond this. Furthermore, whilst the models show some usefulness and can be falsified, they are not predictive (and hence this cannot be considered scientific but remains firmly within the field of philosophy). The reason for this is that in order to generate the graphs and avoid asymptotic behaviour, a definition of commodity is required. The consequence of such is that an activity can only be plotted in terms of relative historical position i.e. after it has become a commodity. This means, all positions of activities which have not become a commodity are uncertain (as per one of the axis of the graph) and therefore approximations. The models do not create a crystal ball and the future is one information barrier we can't get past. Even though the new pattens of organisation are testable it should always be remembered that fitness does not guarantee survival.

That's enough for now, I'll expand the topic sometime later.

OpenStack

There have been many attempts to create open source ecosystems around cloud computing over the last couple of years. Most of them have not fully adopted the largest public ecosystem (being EC2) or been truly open source (instead using an open core model) or they have lacked the experience of large scale cloud operations.

The recent announcement of Open Stack changes this. Entirely open sourced technology for building and running a cloud, supported by an ecosystem of large companies and agencies (including NASA and Rackspace), provision of the EC2 & S3 APIs and the experience of running a large cloud installation. 

This is fantastic news. If you want my view on how this will turn out, well it's rather simple.

OpenStack's move further consolidates the ecosystem around EC2 / S3 which is not only good news for Amazon but also helps propel Rackspace's position as a real thought leader in this space. It's worth noting that the EC2 / S3 API might be supplanted over time, especially as OpenStack builds a marketplace of providers, unless Amazon becomes more open with it. The icing on the cake will be if Rackspace itself (which will use the OpenStack technology) provides the EC2 / S3 APIs, in which case the growth and consolidation around Rackspace's efforts and any providers of OpenStack will become immense.

This is also surprisingly good news for Eucalyptus if they move towards an entirely (or at least more) open approach. In such circumstance, the probability is we're going to end up with a straight forward "clash of the titans" between Eucalyptus and OpenStack to become the Apache of Cloud Computing.

Don't be surprised if Eucalyptus even go so far as to adopt some of OpenStack's work. Marten Mickos is an astute businessman and there are many ways they can turn this to their advantage. However, in general it's not a good time to be any other infrastructure cloud technology vendor, as Simon Crosby makes clear with his "VMWare Redwood = DeadWood" post.

VMWare's position in the infrastructure space is looking unsurprisingly shaky for the future but then they already know of the oncoming disruption, as they made clear with this interview. Why else do you think that VMWare has been busily acquiring into the platform space? RabbitMQ is also increasingly looking like a great purchase for them.

As for RedHat's cloud strategy - they must be feeling increasingly lonely as if no-one wants to invite them to the party. On the other hand, this is good news for Ubuntu, because of both UEC (powered by Eucalyptus) and OpenStacks involvement with the Ubuntu community. Don't be surprised if Ubuntu launches a "powered by openstack" version.

Best of all, it's great for the end users as they will see real choice and further standardisation of a messy industry in the infrastructure space. Of course, the real beauty is that once this happens we can finally start consolidating and standardising the platform space.

Overall, I'm very bullish about OpenStack and its focus on the Amazon APIs. There is a long road ahead but this has potential.

From Slime Mold to Neurons

I wasn't going to write much about clouds, being focused on my new area of research but I could hardly let James' post go unchallenged.

Before I critique the post, I need to go through some basic genetics for those of you who are new to that subject.

DNA is the accepted means of providing genetic instructions used in the development and functioning of all known living organisms. There are exclusions, such as RNA Viruses (which are considered not to be living organisms) and forms of non DNA based inheritance from topology, methylation etc (epigenetics).

DNA doesn't operate in isolation, for example the same DNA sequence in a human produces a multitude of specialised cells. It instead acts in combination with both the environment it exists within and the environments it has existed within. Hence it is more correct to say that DNA contains genetic information that influences the phenotype (characteristics) of an organism.

To keep things simple, I'll ignore the multitude of RNA types (from messenger to transport), the issues of expression, the terminology of genes and 3D geometry and take a few chunky liberties in the description of how DNA works.

In principle, DNA consists of a long double helix sequence of four basic nucleotides (the base pairs) known as C,G,A,T. Different sections of this sequence (referred to as genes) are transcribed and translated into protein structures which affect the operation of the cell. Each three letter word (a codon) of the genetic sequence (i.e. CGT or GAT, giving 64 possible combinations) is translated to an amino acid (of which there are 22 standard).

The entire complexity of life is built upon such simple subsystems which in turn are part of ever more complex systems - cell structures that are part of cells that are part of organs etc. Without this component structure, the level of complexity in living organisms would not have been feasible. It's worth noting that the agility of complex structures to evolve is dependent upon the organisation of their subsystems.

So, what has this to do with cloud?

Well, if you take an example such as Amazon's Web Services, the complexity of the many systems that users have developed with cloud services is based upon the provision of simple, standard subsystems for storage, compute resources and networks.

There is some limited variability in the type of subsystems (for example the size of Amazon instances) and the introduction of a new Cluster Compute Instance but these are the genetic analogy to amino acids which are then used to build more complex protein structures. Your deployment scripts (whether you use a system such as RightScale or another) are your DNA which is then transcribed and translated into the deployment of basic instances to create the complex structures you require.

So, back to James' post. My objection to the post is that whilst you, as a user, can create a slime mould or a neuron or a million other cellular anologies with these basic components, the key is how YOU combine these common and well defined (i.e. commodity-like) components.

James' however infers in his post that we need to see alternative cloud models, not just the "slime mold model cloud" but "more complex topologies" with the "emergence of more topologically calibrated and therefore feature rich clouds". In principle he is arguing for more configuration of these basic subsystems.

Whilst I agree that some additional basic subsystems (e.g. the cluster computer instance) are needed, I'd argue against the principle of wide ranging diversity in the underlying subsystems.  Whilst such a richness of diversity does create benefits for technology vendors - which company hasn't fallen for the "competitive advantage of customizing a CRM system" gag - it will not create the "wealth" and diversity in higher order user created systems but instead lead to a grindingly slow sprawl that will create further lock-in issues, move us away from competitive marketplaces and end up with users spending vastly more time building and configuring stuff which really doesn't matter.

There are always edge cases, however in general the range of subsystems we require are fairly limited and it's from these we can build all the different type of systems (or cells) we want.

If there is anything that should be learned from biological analogies, it is from such "modest entities", such simple subsystems that complexity and diversity is created. We've learned this lesson throughout time, from the electronic to the industrial revolution to the works of Vitruvius.

Unfortunately, as the old adage goes - "the one thing you learn from history, is we never learn from history".

One final note: analogies between cloud computing and biological systems are generally weak at best - my above example is no exception. I use it purely to continue in the same spirit as the original discussion and to try and highlight the core issue of diversity in the subsystems vs diversity in what is built with stable subsystems. I don't recommend comparing cloud computing to biology, unless you want endless arguments.

One very final note: computer models are based on simple arithmetic and hence are bound to Godel's law of incompleteness, neither being being both complete and certain. As activities provided through software tend towards being ubiquitous and well defined, they will tend towards being a good enough component like a defined brick with standardised interfaces. The components themselves will have some inherent non-linear qualities (i.e. the halting problem) which is why the design for failure paradigm is so important.

Biological components are also only linear at a superficial level [e.g. of interfaces such as this codon encoding for this amino acid or a specific cell structure having certain functions or a specific cell type having a defined function] and on an individual level [behind the interface] they are highly non-linear and cannot be described by simple arithmetic means nor modelled with certainty by a computer. For these reasons, biological system have evolved highly complex regulatory systems (such as the human immune system) which even today we barely understand. However, we're acutely aware that a major function of it is to control rogue cells. This level of complexity is far beyond the capabilities of modern computing and also filled with numerous information barriers (the uncertainty principle is only one of many) which prevent us from anything more than approximation. 

However there are many useful concepts in biological systems (Red Queen, Ecosystems, Co-evolution etc) along with certain derived concepts, such as design for failure which have value in the computing world - just be careful on pushing the analogies too far.

--- Update 23 April 2014
Added [emphasis] to clarify certain points

A life less cloudy

For 18 months I ran the cloud strategy at Canonical. It was a real pleasure to work with some very amazing people and play a small part in Canonical's journey. Today, Ubuntu has become the dominant cloud operating system and leads the field in provision of hybrid clouds combining both public use on Amazon EC2 with private cloud technology.

However, I've been in the cloud space for five years, I've got my long service medal and it's time to scratch another itch but I thought I'd wrap up with some general thoughts.

Definition of Cloud

Given that 200 years after the industrial revolution we still can't agree on a good enough definition of that, it was never likely the industry was going to agree on cloud. The problem with cloud is it's not a thing but a transition of certain IT activities from a product to a more service based economy provided through large scale utilities. This change was predicted by Douglas Parkhill in his 1966 in his book the "Challenge of compute utilities". He made the comparison to the electricity industry and talked of a future of public, private, community and government utilities providing computer resource online, on demand, with elastic supply and charged on a utility basis. These characteristics permeate cloud today but just don't try and get everyone to agree.

Why now?

A number of factors were needed before this change could occur. Firstly, you needed the concept which Parkhill kindly provided us with. Secondly, you needed the technology to achieve this service provision but we've had that for the best part of a decade. Thirdly, you needed those IT activities to be well defined and ubiquitous enough to be suitable for the volume operations needed to support a utility business. This has also happened over the last decade.

Lastly, you needed a change in attitude and a willingness of business to adopt these new models of provision. This change in business attitude started with Strassman in the 1990s when he pointed out that spending on IT amounted to little more than an arms race with dubious links to any value created. Nick Carr developed these concepts further and this wisdom has now peculated throughout the business world.

Is everything suitable for the cloud?

The answer to this is "depends". To understand why we must first appreciate that IT consists of many ubiquitous and well defined activities that are suitable for cloud provision. However, IT also contains many new activities that are not ubiquitous nor well defined and hence aren't suitable for provision as cloud services. This of course doesn't mean that such novel activities can't be built on cloud services.

There is a difference between that which can be provided as a cloud service and that which can be built upon cloud services. However, don't get comfortable yet because activities have a lifecycle. Business activities themselves undergo a constant transition from innovation to commodity. In short that which can be built upon cloud services will eventually become that which is provided as cloud services. To make matters worse this is all connected. So activities (even practices) are connected together, they're all undergoing transition from one state to another and this is happening in your business. You can actually map this out (something we did at Canonical) and use this to your advantage but yes, it's often a complex mess.

Benefits & Risks.

The benefits and risks of cloud have been listed countless times before so I'll avoid going through these again and just point out some highlights.

First, cloud is fundamentally about enabling and increasing rates of user innovation (a combination of creative destruction and componentisation). Don't get bogged down into the cost saving arguments because whilst cloud will increase efficiency, a number of factors will explode consumption.

Second, you don't have choice over cloud. Your company is competing in an ecosystem with others who will adopt cloud computing. The benefits of cloud will create pressure for you to adopt in order to remain competitive (this is the "Red Queen Hypothesis")

Third, you do have a choice over how to implement cloud. The hybrid model of public & private service is a standard supply chain trick of balancing benefits vs risks. Expect lots of this and ignore anyone who tells you that private cloud isn't cloud computing.

Fourth, it's not about virtual data centres. This one is a tough nut to crack since lots of people have spent vast sums on virtual data centres and they want them to be infrastructure clouds. The problem is simply one of economics. VDCs are based upon concepts of resilience which have their origin in physical product world and come with a wide range of overheads. Infrastructure clouds are based upon the economics of volume operations with vast numbers of cheap & reliable enough VMs and resilience created through design for failure at the management layer.

Fifth, providers ain't your buddies. In a commodity world of utility services then the market, the exchanges, brokers, standardisation and assurance bodies (providing price vs QoS comparisons) are your buddies. Whilst some providers get this, others will try and convince you of their innovative cloud features. These are the enemies of portability & easy switching and the fastest way to end up paying more than you need.

Six, security, auditing and governance have to change in the cloud world and it's not going to be the same as the perimiterised, defense in depth approach of old.

There's a whole host of topics to talk about from the future formation of exchanges, the creation of rating agencies, the introduction of insurance and derivative instruments, the componentisation effects of the platform space, the growth in importance of orchestration, the …. but this has been done to death for the last five years.

I'll mention one last thing. The secret to Ubuntu's success in the cloud is execution.

We understood our environment and rather than fighting battles which couldn't be won, Canonical choose to adopt a dominant future approach in the market, provide users with choice and real technology and then combine this with a clear story. Hence Canonical launched a hybrid cloud strategy making it easy for users to build with Ubuntu in the public cloud computing space combined with simple and easily installed technology for creating private clouds that matched the same APIs.

By combining these approaches, management tools could be used across both public and private clouds with machine images that could run in both environments. Users had their first real opportunity to exploit a hybrid cloud environment, choosing for themselves the right balance of public and private with an open source technology that limited lock-in and took advantage of both the Ubuntu and the Amazon EC2 ecosystems. Since it's launch in April '09. this approach has gone from strength to strength and has created a beach-head for Canonical to extend into the orchestration, platform and marketplace fields. This is all part of our battle plan and it should be interesting to see how it turns out.

However, the strategy wasn't new, what was critical was the way in which Canonical's excellent engineers achieved this.

Despite the various prognostications on the disruptive nature of cloud, it's old hat. The strategies around cloud were formulated many many years ago. The concepts and ideas around cloud, orchestration, platforms, the economics, the barriers to adoption and how to play in this game are all well known. Even the map of the landscape I used in Canonical to help us understand where we needed to attack was based upon an earlier version from Fotango. Nothing is that new. 

If you're looking for that great strategy to bring your company into the cloud, be aware that whatever you think of is almost certainly being done by others and has been contemplated to the nth degree by many more. Today, success in cloud is all about partnerships, execution, tactical moves and timing in a great game of shogi that is already well advanced.

The time for strategists in the cloud space is over and it's all about how the game is played.

Don't get me wrong, the game is still an exciting place. There are all manner of tactical plays, twist and turns and efforts to help standardise and form this future industry. However, strategy is my passion, it's where my heart lives.

Which is why I have joined CSC's Leading Edge Forum to pursue a new challenge at the very centre of change. There's a number of (what I consider important) experiments I need to run on organisational change and this is one of those rare opportunities I'll get to make it. Change and I have some unfinished business.