Open Source and the Cloud

"It's important to understand that open source in this utility computing world is not a tactic, it's not a strategy, it's the only practical way of creating and competing in a marketplace"

OSCON, 2007

I've been asked why the OCI principles has the phrase "and at least one such implementation must be licensed in its entirety under an Open Source Initiative (OSI) approved license"

First, it's important to understand that in a utility world, any standard must be running code for reasons of semantic interoperability. In order to create a competitive market rather than a captured market this has to be open source. You can find more on my discussion of competitive markets and open source in various of my talks and posts between '06-'08. However, you'll find the basics in the OSCON 07 presentation.

By implementations this can refer to multiple providers offering the same code base or multiple providers offering different code bases. However, as long as all the implementations are "full, faithful, independent and interoperable implementations" with the open source implementation then you have a free market. The open source implementation in effect acts as the reference model.


For example, multiple Cloud Foundry providers means multiple implementations and there already exists an open source implementation in Cloud Foundry itself. Of course, providers could set up with modified versions of Cloud Foundry (e.g. operational improvements) and as long as they are full, faithful and interoperable implementations they would meet the OCI principle.


You cannot avoid the need for an open source reference model in the OCI principles. But that's because you cannot avoid the need for an open source reference model in a competitive utility computing market.


Competitive markets and Open Source are intertwined. Always have been. It's what we tried to do with Zimki all those years ago.

Dante's Quadruple circle of hell - The Expense System

I have a fervent dislike of expense systems. It's not that I have particular dislike of an expense process but it's the systems themselves that always get me.

First, a good example (as in one I like) of an expense process.

The company gives me a company credit card which the company owns. There's a limit on the card. I spend as is required on a principle of trust. The company accountants receive the bill. If anything looks odd they come and ask me.

This is my favourite expense process, it's blissfully happy. It works on trust as in the company trusts me and gives me this right. It's hassle free and occasionally the accounts team might ask what this or that item was for. It works on an exceptions basis.

The few times I have experienced this process, I've been delighted.

Second, a bad example (as in one I hate) of an expense process.

The company gives me a company credit which I own (i.e. it's my money). I spend as is required, in other words I dip into my own pocket and spend on company stuff.  Ok, it's called a company credit card but it's actually my money.

Then after spending my money, I have to fill in forms to claim my money back. There are rules, usually lots of them and often discussions about whether this item is valid or not and of course I have to have receipts.

I usually have to fill in a horrendously detailed system, upload scanned images of receipts, comply with the rules, deal with any discussion - which all takes up my time. Along with this, there is often any number of other authorisation systems which have to be complied with in order to get back my money which I've in effect lent the company. Of course any late charges are my responsibility and there's no question of interest being paid.

The many times I have experienced this process, I've hated it.

In fact, I normally start to respond at a subconscious level by avoiding any expense. I start to realise that things I would normally do, I avoid. It's the equivalent of being invited to a pub by a friend, them asking you to pay for the drinks but they'll pay back later and when it comes to later they ask for itemised receipts and start questioning. The chances of me being available next time I'm invited drops quickly to zero.

I do find however some people love it because of some other benefit like they gain air miles on the card or some other point scheme. As someone who has taken a thousand flights in their life - the last thing I ever want is air miles.

So why am I mentioning this?

Well expenses for me can often be a time of increasing expletives, frustration and the thumping of a keyboard. But, it turns out my uncle enjoys the experience and even has an expense afternoon. A part of his week he sets aside for doing expenses routinely.

However, he does feel a bit guilty about this because he feels it's not proper work.  Well I agreed with him. In my experience filling out expense systems is not proper work, it's in fact hell and as such should count for quadruple time.

But I'm curious, do other people feel guilty about using work time to fill out expense systems? Why?

I don't like Hayek vs Keynes

I'm pro Hayek, pro Keynes and very much Anti Friedman. I see an awful lot of discussion which is Hayek vs Keynes and I don't quite understand it. Certainly, these two great thinkers had disagreements and their approaches (one top down, the other more bottom up) were different but their models are not mutually exclusive. It's the mix of Hayek, Keynes and Schumpeter which has always made economic sense to me.

To explain why I feel this way, I'll use four graphs. The first two examine the process of evolution as opposed to diffusion, the third is an examination of value chain vs state of evolution and the last is one the cycle itself.

As activities commoditise they can enable higher order systems to appear e.g. electricity begets radio, computing, Hollywood whereas nuts and bolts beget machines which in turn beget generators etc. These higher order system are where new value is generated and hence we see a flow of capital from past industries (which are commoditising) to new industries (which are being created).

The process of evolution is not however smooth as companies build inertia to change (due to past success), which is why when an activity commoditises it's often associated with new entrants into the market and disruption of existing players.

Figure 1 - Evolution & Diffusion

Figure 2 - The Process of Evolution

Figure 3 - Evolution vs Value Chain

The above can be simplified to a cycle of commoditisation enabling the genesis of new activities (which we often call innovation but there's problems with that specific term). The shift from product to commodity and utility services and the associated inertia barriers to change mark a phase change in an industrial ecosystem. We go from a relatively peaceful state of competition with sustaining change exceeding disruptive to a fight for survival, with new entrants and disruptive change exceeding sustaining. We literally go from peace to war.

Of course, the explosion of new higher order systems (due to componentisation effects) create a rapid growth of new industries (genesis) and a time of wonder and amazement. This cycle of wonder, peace, war repeats throughout history both in specific industries and at the macro economic scale through Kondratiev waves or what we more commonly called ages.

The age of electricity started not with the genesis of electricity provision (which can be traced back to  Parthian times) but with Westinghouse & Tesla and utility provision of modern day electricity supply. This disrupted past industries (war), created an explosion of wondrous new activities built upon this component supply (growth) and eventually the new giants of industry settled down (peace). I've created a simplified map of this onto figure 4 to show the cycle (ignoring the value chain).

Figure 4 - Cycle of Change.

So, now to Schumpeter, Hayek and Keynes. The overall cycle describes concepts in Schumpeter's work on business cycles. During the growth and peace cycles, the application of Hayek's policies seem the most logical - limiting excessive growth in credit and the impact of central bank policies such as low interest rates which encourage speculation, lower savings and bubbles etc.

However during the war phase (where much of past industry is suffering inertia often compounded by the demands of financial community) then direct investment in new industries and the stimulation of demand - a Keynesian approach - seems a more effective way of balancing this market change.

But this is no different to the issue of managing activities which due to changing characteristics vary from agile development (in the genesis phase where they are more chaotic) to six sigma (in the commodity phase where they are more linear). There is no one size fits all policy and instead you need to use the right policies for the right stage of evolution.

This is why I don't get the whole Hayek vs Keynes debate - which noticeably is a very US based discussion and occurs to a much lesser extent in Europe. The debate is as simplistic as the whole Agile vs Six Sigma, SQL vs NoSQL, Push vs Pull, Hierarchical vs Networked debates of one size fits all.

The answer always seems to be both are right, you just need to know when to use each. So, I'm pro Keynes and Pro Hayek as there is both a time to steer markets and a time to set them free. Furthermore, since markets evolve at different rates then you need to use both approaches at the micro level i.e. one market you let run free whilst in another you can be busily investing and stimulating in at the same time.

Does this make me Pro Friedman? Absolutely not. I'm Anti Friedman and I have little time for the Big Dogma of Monetarist theory / Neoliberals which has been created in his image. I do understand the political reasoning of supply side fiddling with QE because it helps maintain the illusion of house and stock prices (through cheap foreign capital) but continues the stagnation of our real internal industry and society. It is not worth it.

We need to direct our industries to overcome their inertia and that requires direct investment along with stimulation of demand. I find scant evidence that the financial community can resolve this in a timely fashion as they have inertia of their own. Instead, China's huge investments (at this moment in time) in internet technology, renewable energies and manufacturing technologies such as 3D printing & printed electronics suggests an alternative way. They seem to know how to play this game and they're aiming to grab the future.

We should have buried the monetarist dogma long ago and started actually playing the game. The whole Keynes vs Hayek debate feels like just another artificial campaign to maintain the status quo and all the suffering that comes from it (social inequality, lack of true mobility etc). I, for one, don't like it.

On Predictions

There are three basic forms of predictions that I make, which given this recent podcast I thought I'd better make clear.

First, there are predictions I make regarding the overall process of evolution and the cycle of change. These are based upon causation, correlation and past testing and I'm confident enough in this to describe the model as a weak hypothesis.

Second, there are predictions I make regarding specific industry actors. These are based upon my opinion and naturally any outcomes depends upon the strategy of the actor, the impact of others and economic events. These predictions are therefore inherently uncertain. I'm not confident in these at all as they are purely conjecture that I consider likely.

Lastly, there are predictions I make regarding my on going exercise of prediction testing. Here I'm deliberately aiming for a 50% prediction rate of the headline predictions which are based upon a hundred plus components. This is purely an experiment I'm running to see if I can improve some general models I have. Overall, this experiment is simply an idea at the moment which I'm refining.

For reference, the general scale I use is as follows** -

Idea = anything you think of. It's an idea. No Confidence.

Conjecture = idea + some initial supporting data (upto a hundred or so data points) demonstrating merit. Extremely Low Confidence

Reasoned Hypothesis = idea + initial data + correlation (through a couple of experimental observations) + initial models of causation. Low Confidence

Weak Hypothesis = idea + data (a few hundred to a thousand or so data points) + correlation (supported by a couple of experimental observations) + causation + verification (either independent confirmations or a couple of prediction tests). Weak Confidence

Strong Hypothesis = Weak Hypothesis + large volume of confirming experimental data (tens of thousands of data points) + correlation supported by significant independent experimental observation (in the order of hundreds of) + verification (peer review and and hundreds of prediction tests).  Reasonable Confidence

Theory = Strong Hypothesis + significant volume of data (hundreds of thousands of data points) + independent verification of experiments (in the order of many thousands of) + widespread acceptance by scientific community + large volume of prediction tests (many thousands of). High Confidence

Universally Accepted Theory (law* in all but name) = Theory + so much data (many millions of data points) + so much independent verification (hundreds of thousands of independent experimental observations and predictions tests) that it has become universally accepted within the community, we've almost stopped asking the question and will be genuinely surprised if someone shows it to be wrong. Extremely High Confidence

* In the normal sense a scientific law is simply a description of observed phenomenon whereas theory is the explanation of those observed phenomenon e.g. Newton's Law of Gravity is not the same as the Theory of Gravity. However, I don't know an appropriate word to describe a universally accepted theory, the temptation is to call it a 'law' except that creates all sorts of other confusion. So, I'll simply note that there are categories of theory which have become universally accepted. 

** I refined this list on the 17 August 2013 to give some examples of volume of data that I'd expect to see.

A Tale of Two Systems in Eight Magic Quadrants ...

After speaking with a friend of mine Ed Meagher, I bring you a Tale of Two Systems in Eight Magic Quadrants ....

Bears vs Salmon

Vendors vs Government.

Interesting moves by VMware

Many years ago I made a prediction that VMware (or more specifically its master EMC) would focus on an open source platform play and look to monetize its existing hypervisor based business.

The latter part I suspected will occur through an IPO of VCE or some equivalent move. The reason being is that at this moment much of the buzz in financial analyst circles is around hybrid cloud environments consisting of a combination of private & public infrastructure and VMware's technology is touted as well placed for this industry.

What the analysts don't seem to understand is that this hybrid model is purely transitional and will be replaced in the majority by a hybrid model of public and public i.e. use of multiple public sources. The private enterprise cloud market is at best transitional and has a limited lifespan until it becomes niche.

Still, there's nothing like selling at the top of the market an area of technology which you know will ultimately be disrupted. If the analysts help you do this, bully for you.

However, EMC is very astute. There's an awful lot of value which can be gained by providing a commodity infrastructure services through the development of a wide ecosystem and the exploitation of such through an ILC (innovate-leverage-commoditise) model.

So, I've often toyed with the idea that VMware (EMC) would :-

1.  Develop and launch on an open source platform play. This is a key part for the future in terms of developing a competitive market of providers, destroying any competitor's ability to differentiate in this space and enabling VMware to exploit this space with a range of higher order services (see management tools).

2.  IPO or sell the existing hypervisor technology. This is to maximimize the value of a range of technologies which will ultimately be disrupted and provides a neat way of overcoming inertia to change within the organisation by spinning it out of the company.

3.  Develop and launch a range of cross public cloud management tools. This would be part of a future plan of creating brokerage, exchange and assurance capabilities across both infrastructure and platform and monetizing these layers of the computing stack.

4.  Launch a major open source IaaS offering.

My thoughts on this ...

So, the first part and the development of an open source platform play occurred a couple of years after I made the prediction. From all reports, Cloud Foundry is making great progress and there are multiple providers being set up from ActiveState, Uhuru Software, PaaS.io, Tier 3, AppFog and VMware's own service.

The second part of the prediction I consider likely to happen in the near future and I view that VCE is the likely vehicle for this. It's critical that any IPO is achieved prior to any market decline and the subsequent questioning over the future of private enterprise class clouds. It's not that they're not viable, it's just you have to run them with a brutal commodity focus to make them operationally efficient.

I've seen examples of private clouds operating at a third the price that AWS charges publicly on a like by like comparison of default EC2 instances. Which is why I'm one of those who argues that AWS is probably operating at a very high margin (>60%) and there's a lot of scope for price reductions.

I believe there is a reason why AWS doesn't simply reduce its public pricing drastically and that's because I suspect that AWS has a bottleneck - buying up enough land to build data centres to keep up with its rapidly expanding growth. Dropping prices aggressively will just increased demand and exacerbate this problem. Any price reductions will have to be carefully managed.

This is also why I think that the GCE (Google Compute Engine) focus on initiating a price war is smart because whenever you face a competitor with a bottleneck, creating a price war to increase demand beyond their ability to supply will naturally fragment the market.

I said the same to IBM, Dell and HP back in 2009 (when I was with Canonical) and told them it was in their interests to move quickly and create a price war with Amazon unless they wanted to end up being in a strategically weak supplier position to a few dominate cloud players. I don't think any of them listened but that's not my problem, that's theirs.

Anyway, to operate at the levels to be operationally efficient it is going to be difficult if not impossible when you use proprietary licensed software. My models reckon you can afford to spend a couple of dollars (i.e. $1-3) per year per EC2 equivalent on software licensing & software maintenance. We are told that vCloud Director is "non-existent" in the service providers space, this doesn't surprise me.

I don't see a big future for these software products in the public service provider space and I only see a time limited future in the private enterprise space before it becomes niche. Hence IPO before the trouble hits is a smart play in my book and EMC does tend to act in a really smart fashion.

However, this leaves two others parts of the prediction which I've not made public before especially as the open source play I've considered to be too bold a move even for EMC. I'm now coming to the view that I've underestimated their boldness.

My view here is fairly simple. Once an IPO had been achieved (or some other mechanism of separation) then a fully fledged IaaS attack could occur. I've been watching this space with interest looking for hints that such a change might be coming which is why I'm increasingly confident that this is a possible game they might play.

First, I noted that VMware / EMC joined the Open Compute project which if you're going to get into the game of building massive scale and efficient commodity based data centres is where I'd start.

Next, I noted that Paul Maritz has been promoted up to running strategy for EMC itself, which in my view is a shoe-in eventually for the CEO role.

Then, I noted that VMware acquired Nicira.

Finally, I heard about project Zephyr and how VMware (i.e EMC) was willing to attack the IaaS space. This was enough to make me publish this post.

Now, if I was going to play a major open source IaaS game then I'd probably base it around Cloud Stack (Citrix's donation of $200m worth of cloud technology to the ASF) and combine this with an S3 equivalent and software networking capabilities. For the S3 equivalent I would probably acquire Basho for Riak CS and then add this into the Cloud Stack project under ASF.

If I then had within the company the following skills - hypervisor technology, large scale efficient data centres (i.e. open compute), distributed storage (Riak CS), software defined networking (Nicira), management tools (Dynamic Ops) then I'd be almost ready to play a major open source IaaS play.

The other bits I'd need are some partnerships on highly commoditised hardware (I'll note the recent partnership between EMC and Lenovo), acquisition of large data centre space (Project Zephyr) and skills with Cloud Stack (which VMware already seems to have).

Of course, Project Zephyr will be touted as selling existing technology but if I was running this play then that would be just be marketing cover for the hidden agenda. And to be truly Machiavellian, given that there would be an inevitable reaction of existing partners to Project Zephyr, then I'd use this as marketing cover for the reasons why I'd IPO VCE - an argument of separation of concerns.

So in my play, into VCE would go the existing hypervisor business (along with all the inertia it creates to change) and then this would be IPO'd in order to "avoid the channel conflict" that Zephyr creates with partners and Nicira creates with Cisco. In one swoop I'd have maximised value (through IPO) on a business that I think will be disrupted in the medium term and kept all the components that I think are the long term future by using a completely plausible story of channel conflict. The capital I'd have raised on an IPO would be ploughed right back into an open source IaaS play which I would then reveal.

At lot of what is needed to make this happen appears to be already in place, so as completely far fetched as it sounds, I'm expecting in under 18 months for EMC / VMware to have acquired Basho, launched an open source IaaS effort and IPO'd VCE.

Of course, I have no inside knowledge of EMC / VMware strategy. Their plan maybe something else but this is what I would do. If if happens and it succeeds then this will be a truly outstanding example of how to deal with inertia and the threat of disruption.

[Update 15 Mar 2013]

First, VMware is pushing heavily on the hybrid model which is all par for the course and the open source platform plays have been made and pushed into a new spinoff known as Pivotal Initiative (under Paul)

Second, EMC itself appears to be scouting out some more commodity IaaS providers which makes perfect sense given the above.

In my original version, EMC would have pushed the "to be disrupted technology" into VCE and flogged it off at the top of market whilst keeping the interesting future technology within VMware. The excuse for flogging VCE off would be a channel conflict because VMware would make a big commodity IaaS play.

However, given that EMC has pushed the interesting future technology into the Pivotal Initiative, leaving the "to be disrupted technology" inside VMware then VMware itself can be flogged off (i.e. you don't have to use VCE to create the separation).

However, you still need the plausible excuse of channel conflict and a way of making a big commodity IaaS play. Which is why I think the EMC interest sounds reasonable.

So rather than:-

• "to be disrupted technology" into VCE (EMC exit)
• "interesting future technology" in VMware (EMC keep)
• VMware to play a commodity IaaS play  (EMC keep)
• VCE ("to be disrupted technology") is then flogged off (EMC exit)

An alternative route would be:-

• "to be disrupted technology" in VMware (EMC exit)
• "interesting future technology" into Pivotal Initiative (EMC keep)
• EMC to play a commodity IaaS play (EMC keep)
• VMware  ("to be disrupted technology") is then flogged off (EMC exit)

The effect is the same, EMC keeps the interesting tech (open source platform play), builds a commodity IaaS play and flogs off hypervisor technology for a good price. I was a bit surprised that DynamicOps etc remained within VMware, this seems a curious decision.

Will it pan out like this? We shall see. Whilst overall trends (e.g. evolution from product to utility) are highly predictable, the actions of individual actors aren't.

Self disruption and super linear ...

On Self Disruption

Back around 2005, in the days of Fotango, we had started to introduce a structure which I commonly refer to a Pioneer, Settler and Town Planner. This is structure which is not by type (i.e. IT, Marketing, Business Development) but instead by Flow (from chaotic to linear, from genesis of an activity to commodity).

Under the structure, Pioneers are rewarded for experimentation and the genesis of new activities. Settlers steal successful patterns from the Pioneers or any wider ecosystem to create more defined components, a library of useful things. The Town Planners steals successful components from the Settlers to create commodity components through utility services with a focus on automation and standardisation.

The advantage of the structure is that each group feeds of the previous with Settlers stealing from Pioneers and Town Planners stealing from Settlers. The circle is completed because the Pioneers build on the services the Town Planners create.

Now back in the days of Fotango we started with IT and called these groups development, framework and operations however we had already started the process of folding business development and marketing into this structure. I know of three other companies that use such a structure, alas it's not enough to test the findings conclusively.

That said, the early findings included a remarkable surge in efficiency and rate of genesis of new activities. This combined with our creation of a platform underpinning this concept led to some stunning results including the development and delivery of a prototype wiki with client side preview from concept to live on the web in under 53 minutes.

However, the real beauty of the model is that it's self disruptive due to the structure creating a pressure for each group to steal from each previous group. The creation of such a pressure overcomes inertia which may have formed through past success.

On Super Linear

Our focus at Fotango was also on the development of ecosystems hence our entire platform was provided though "open" APIs back in 2006. We had several years of experience in building with web services and we intended to open source all the components in 2007 to create a competitive market with a view of establishing higher order systems such as exchanges, brokerages and assurance.

Alas, there's another story here of political capital and inertia within large companies and since this is well documented, I won't go there again.

The model which I use to describe the goal of an ecosystem approach is known as Innovate-Leverage-Commoditise and its use is rather simple and becoming more common today.  First, you start by providing more linear activities i.e. those suitable for utility provision because they're widespread and relatively well understood. Examples include compute resource, storage, databases etc.

The purpose of these is to allow an ecosystem of other companies to build on top of your service and to encourage those companies to innovate (as in genesis of new activities) by reducing their cost of failure (through provision of utility services). In effect, some of those companies you can consider to be an extension of your own Pioneers.

As any new activities spreads through the ecosystem, you should be able to detect this change in consumption of APIs. This is the Leverage component of the model as you use the ecosystem itself to do the pattern spotting and it in effect operates as an extension of your own Settlers.

One a pattern is spotted, you commoditise this to utility services. Yes, you will be accused of eating the ecosystem, though you can always control this messaging by balancing acquisition vs copying. However, each time you eat the ecosystem, the new component services you create should help grow it. This balancing act is critical to keeping the ecosystem healthy.

As a company, you will appear to be highly innovative (but in reality others are creating those new activities), highly customer focused (but in reality the ecosystem is telling you what you need) and highly efficient (because you're focused on utility services).

Bye, bye the old mantra of choose one of these strategies. That's been a busted flush for many years.

Now, as a very weak hypothesis the rate of innovation (genesis), customer focus and efficiency can increase almost linearly with the size of the ecosystem, though the volume of data is not strong enough to publish this (unless we're talking about various popular management publication in which case any old nonsense seems to get through).

The beauty of this is that ecosystem growth can be super linear with the actual size of the company and hence if the hypothesis holds then it's possible to become super linear for innovation, customer focus and efficiency through an ecosystem model. Given what I know, this appears to be the case but it's not conclusively shown.

Whilst competitors have built self contained Towers, Amazon has built a city with itself at the heart of it. The old model of big companies are less nimble is a busted flush here because genesis of activities and agility extend into the surrounding ecosystem.

On Self Disruption and Super Linear.

It's perfectly possible to organise a company using an ecosystem model such Innovate - Leverage - Commoditise around a structure of Pioneer - Settler - Town Planner.

In this case the Pioneers works on encouraging and creating new activities - a mix with a heavy emphasis on Community Evangelists.

The Settlers work on identifying the new patterns in the ecosystem and exploiting this - a mix with a heavy emphasis on Data Scientists, Strategists and Business Development.

The Town Planners work on commoditising new patterns - a mix with a heavy emphasis on Engineers and Operations.

There is no reason that I'm aware of why a company can't be both continually self disruptive and superlinear for innovation, customer focus and efficiency. In fact, the only reasons which I know that prevent this is the poor use of ecosystem (if at all) and poor structure (e.g. organisation by type).

I suspect that Amazon will in the future be cited in endless "management" books on why these problems can be solved. This is also why I've said many times over the years that Amazon is likely to be that $1 trillion market cap company and it will become far more powerful the bigger it gets.

Why the Rant?

I'm tired of people telling me that you can either be innovative or customer focused or efficient but you can't be all of them.

I'm tired of people telling me that companies can't be super linear for the above and twisting Geoffrey West's work as evidence of this.

I'm tired of people telling me that companies can't be self disrupting and that it's inevitable that companies will be disrupted.

The above statements only hold if you build your company with management practices from the last Century.

Open your eyes, the software world is already starting to operate in an environment where all of the above is possible and this will soon extend out into manufacturing (though commoditisation of the manufacturing process).

Get used to it and start thinking about how to do this better than competitors.

Rant over ...