Bits or pieces?

Stop Online Piracy, NOW!!!!

2012-01-24T09:37:00.003Z

If Congress wants to stop online piracy, there's another way. Ban all content which is not creative commons or equivalently licensed material (e.g. GPL) from the internet. Any infraction should be treated as other security violations and made the responsibility of the copyright holder for not taking enough security measures to ensure that their content never reached the internet.

Won't that destroy online films?

Of course not, people will still want easy access to content from a convenient and trusted source (such as NetFlix) and where there's demand, supply will follow. The trusted brand is all important and people will still pay a reasonable subscription for it. The content producers will just have to adapt to a world where they can still make money from abundantly used creative commons licensed content rather than scarcity.

Won't that destroy online journalism?

Of course not, people will still want easy access to content from a trusted and respected source which provides an analysis of what is happening. The trusted brand is again all important and advertisers will still want to promote their messages. Certainly, people will copy it, so you'll need to build a loyal following and look at those copying sites as free marketing.

Won't that destroy the online music industry?

Of course not, people will still want to hear their favourite bands live, attend gigs, buy merchandise from the band site and they'd still pay (either through subscription or advertising) for a trusted and useful service. Certainly others will copy it, so you'll need to do the usual - build a loyal following, focus on creating a strong trust relationship and think of those copying sites as free marketing.

Won't that destroy the past models of media?

Well the media industry always cries wolf over piracy and change but certainly forcing them to keep copyrighted material secure will make it difficult for them to distribute. However, since many don't seem willing to adapt to a new world (hence SOPA/PIPA) and the cost of introducing legislation to protect them will harm more future focused industries (i.e. the internet is a component of these) then maybe legislation may be needed to force change.

Even if the content is freely available, I'd still pay to have easy access to it through a useful and trusted service, I'll still want to attend gigs, attend lectures, buy merchandise (even books) and watch movies that include product endorsements etc.

Yes, it will be a different world but either the traditional Media companies adapt to a world where you make money from abundance or the US will need to sacrifice its future competitive position (by harming its internet industry).

It's not like those traditional companies haven't had almost two decades to prepare for this change. What have those media executives been doing - playing golf with Kodak? What did you honestly think that digitisation and the internet was going to do - increase your profits by reducing your distribution costs?

Banning copyrighted material is an extreme option and I do believe in a more balanced approach. However, if the traditional Media industries are going to try and use legislation to avoid change, then someone needs to start thinking about how to use legislation to force them to adapt. Otherwise the US tech companies will be constantly in a defensive, rear guard action against acts like SOPA / PIPA etc.

The best form of defense is a good offense.

Reprinted from G+.

Mystic Me 5.0

2012-01-17T13:08:00.019Z

I'm a bit late with my predictions for this year as a couple of components have already started and so I've had to re-write those parts. However better late than never and in any case this is mainly for my testing purposes. The normal rules apply, the predictions are built up of individual components and each and every component must be correct for the prediction to be correct.

Since, I managed to overshoot my target for last year, I've made the predictions even more detailed using well over 100 different components. I'm aiming for a 50% target (not higher and not lower) which is the ideal balance between usefulness and accuracy.

So, without any more egging of the custard, here goes :-

Predictions for 2012

Cloud:
Open stack will gain further momentum in the popular press with multiple providers coming online to form a fledgling market. However debate will intensify over the wisdom of providing multiple APIs and whether Open Stack should focus more on being an AWS clone due to evidence of the growing success of Eucalyptus. This situation will be further complicated by Amazon launching a managed "Data Centre in a Container" product aimed at at large enterprises as an onramp to use of public AWS services.

The adoption of cloud computing will continue to outstrip earlier analyst predictions and pundits will cite AWS as exceeding $2 billion in revenue.

The confusion over "enterprise clouds" will grow due to marketing efforts promoting Enterprise Class vs Commodity based clouds, however there will be a backlash including some high profile customers declaring them as of dubious value. Platform as a service will have a strong year with CloudFoundry in particular growing significantly in both community involvement and media coverage.

There will also be no let up in the pace of mergers and acquisitions in this industry with a particular focus on Devop and Management systems. Both ARM and Ubuntu will strengthen their positions in the cloud space. In particular, we will see increasing mention of a standard computing stack involving Ubuntu, OpenStack and CloudFoundry.

Countering these developments will be an increased involvement of Gov bodies with the view of introducing legislation to the cloud with licensed cloud operators.

Big Data will continue to rapidly grow in prominence, however the focus will switch more towards utility provision of big data systems and the importance of algorithms. In particular, data competitions will have a strong year and it will become increasingly clear that they are used not only for improving algorithms but as sources for recruitment of talent.

There will be a number of high profile articles questioning when (not if) cloud will dominate financial ERP and more traditional Enterprise spaces with popular wisdom shifting towards the near future i.e. less than 5 yrs. However, what won't be clear in the first half of the year is which companies will dominate this space and instead concerns will be raised over whether existing software vendors can overcome internal inertia. By the end of the year, it will be clear that an outside player will dominate.

Environment:
Total Arctic Ice volume will decline to the lowest level on record raising concerns that a tipping point has already been reached. The melting season will be considered to have extended again and the UK will suffer one of the most severe winters on record. Despite the unpopularity of nuclear power, there will be a number of high profile environmental articles highlighting it as a necessary evil in terms of combating climate change.

Economy:
Despite assurances by the BOE (Bank of England) that inflation will reduce by the second half of the year, RPI will have increased on a year by year basis. The technical recession in the UK will turn into a full blown recession with increasing discussion in the BOE for another round of quantitative easing. Interest rates will be kept at their current historic low. The driving forces being many of the UK actions will be from Europe.

In detail, the sequence of events include :
In the first half of the year there will continued uncertainty over the European debt crisis and exposure of banks to financial instruments based upon this. There will be increasing calls for the ECB to act as the lender of last resort and underwrite individual countries debts across Europe but the ECB will initially refuse.

With increasing social pressure within European countries, the core group of Europe will take drastic action. First, selected countries will default on the Gov debt but remain within the Euro causing increasing market reaction to Euro debt, weakening of the Euro and strengthening of UK gilts and GBP. As GBP strengthens, the FTSE will weaken (as foreign capital seeks to take profit) and to counter this the BOE will embark on a significant round of quantitative easing, possibly in excess of £500 bn depending upon how insane they are.

At this point, the ECB will step in and consolidate the remaining Gov debt across the core Euro group into EuroBonds and act as the lender of last resort for future debt. Investors and rating agencies being caught flat footed will cry foul, however the strengthening of the core Euro group will cause both the Euro to rise and the debt crisis to recede in Europe. However, investors in those selected countries which have defaulted (i.e. banks etc) will be left with a realisation that they have been gamed. Legal actions will result but losses in those investors will be enormous.

In the core European countries, those banks most impacted will be nationalised and it will become increasingly clear during the year that this had been planned as the most significant damage will be felt in the city of London. By the end of the year, the core Euro groups will start to re-define the single market agreement to be limited to a Eurozone. Faced with mounting debts, weakening of the financial market and the strengthening of the Euro market, the FTSE will start to fall significantly by the end of the year. The BOE will then again raise the spectre of even more QE.

Society :
There will be continued protests in Europe over austerity measures in the first part of the year, however by the end of the year these will lessen. In the UK however the reverse will happen.

Protests and strike action in the first part of the year will be mild, however by the end of the year with a massive increase in gov debt (due to QE3), rising inflation (due to QE3), weakening internal economy (due to QE3), isolation from Europe (due to Euro core and ECB action), further bail-out of the banking system (due to Euro core and ECB action) and increasing government austerity ... the mood will darken considerably.

Someone, somewhere will write a popular but ridiculous press article on whether this is the "End of Britain?" By the end of the year, one member of the MPC will write an article explaining their concerns that they have got it wrong.

Politics :
The year will show increasing tension between the coalition partners with a number of high profile spats. Despite this and with press pundits predicting a collapse of the coalition, the darkening public mood will convince both parties that an early election will lead to a rout. In their own interests, not in the interest of the country, no early election will be called.

Despite the calls for more legislation of the internet and protection of vested interest, there will be a growing realisation that the UK must embrace a future which is not the past. Such calls will find a champion and increasingly the Government will talk about the end of large scale IT projects, embracing a more open future and a need for industry to adapt. Increasingly measures will be discussed to encourage high technology start-ups, to support open source, to end current Government purchasing practices and to limit IP effect on the wider industry.

By the end of the year, despite the poor economic situation, the first glimmers of a bright future will appear as a number of high technology companies will openly discuss moving to the UK.

Technology:
VMWare will clearly act as two operational divisions - one focused on infrastructure, the other on platform whilst VCE will be touted as a potential IPO for 2013.

The dominance of Android on phones, tablets and TVs (as demonstrated by market share of units shipped) will increasingly raise speculation over Apple's future with several popular press articles asking whether this is a re-run of the Mac vs IBM PC.

Amazon will also have an exceptional year in sales of their tablets with Amazon and Samsung being seen as as the two dominant players (by volume of units) in the tablet space.

Google TV & Google Wallet will exceed expectations with G+ continuing to grow rapidly exceeding 350 million users by the end of the year. Despite hostility to the integration of G+ with Google Search, and the potential dangers of data gravity effects, Google will not discontinue the effort. As a consequence Facebook will buckle and start to adopt a more open approach to data.

Early in the year pundits will discuss the potential for Twitter as 'the' worldwide real time messaging system and there will be, at least one, high profile attempt to acquire it. Twitter will both refuse acquisition attempts and refuse to integrate into G+. By the end of the year the pundits will change tune and articles will question whether Twitter is a "dead man walking".

Mobile banking will also have a phenomenal year with pundits speculating whether this is the beginning of the end for many traditional retail banks.

Media :
Despite early success, 2012 will be marked by an increasing "war" between past industries and the future. The battle of IP will cover many fronts simultaneously including legislation through Congress, further introduction of DRM on devices and attempts to reinforce Global IP laws.

The battle in the US will turn particularly bleak as media companies aggressively fight a campaign through traditional media channels including assaults on the characters of many public opponents. The pro IP lobby will also find an unlikely ally in China. Due to expansionist policy, Chinese backed companies will increasingly become active in US IP law and provide funding to increasing IP / Copyright legislation. Opponents will highlight how China's policy in the US differs from its own home policy which will not strengthen such laws.

Despite vocal public opposition the US Congress will continue to enact PIPA or an equivalent. By the end of the year, a number of leading Internet companies will have openly raised concerns that they may need to move out of the US. Media companies which have adapted to the new environment, such as NetFlix, will continue to grow rapidly but increasingly will find themselves dragged into the political battle with content providers.

At the very end of the year, a high profile article (probably HBR) will be written concluding that the US has just handed the future of the internet to other nations.

Manufacturing :
Both 3D printing and printed electronics will have a robust year in terms of growth and public awareness. There will be a marked rise in start-ups and funding in this space, with numerous public articles describing the technology as the future of manufacturing.

The first hints of hybrid printers (both electronic and physical form) will surface along with technology articles questioning whether new forms of computing language will develop covering both physical and digital function.

None of the major printing companies will make significant moves into this space during 2012.

A number of articles will also raise concerns on the issue of security and whether this technology will lead to widespread piracy. These articles will conclude with the importance of DRM and a fledgling lobbyist organisation will form to promote these concerns in the US.

Things to Watch :
The key watch words of 2012 are Ecosystem, Openness and "Do it yourself" IT.

Increasingly there will be a clear separation between traditional organisations and a new form of next generation companies. Key characteristics of this next generation which will be highlighted throughout the year include :

use of cultural strategy
cell like organisational structures
use of platforms to develop ecosystems with competition based upon ecosystems
use of commodity components in IT
emerging architectural practices (design for failure, chaos engines and distributed systems)
extensive use of analytics and algorithmic regulation
intensive focus on strategic gaming in competition
use of open source as a tactical weapon against competitors
a focus on disruption of existing industries as opposed to profit or expansion into geographical emerging markets.

MISOG's :
Despite the best efforts of the Olympic Committee there will be endless grumblings about the Olympics in London covering the cost, the legacy, the exclusiveness of the event, ticketing, transportation issues, failure of IT systems, unpreparedness, rising costs of rent and excessive security. There will be some protests over the event with increasing concerns that the project costs have overrun and the UK could ill afford the event. Much of this will be couched in terms of the increasing economic gloom in the UK which will overshadow the event. Despite this, the UK will have a good event.

An open letter to Congress on SOPA / PIPA - from Aliens4SOPA

2012-01-17T12:21:00.003Z

To understand the impact of SOPA / PIPA we need to get rid of some very basic misunderstandings.

First, explosions of industrial creativity DON'T follow the invention of a technology but its commoditisation i.e. it wasn't the invention of electricity but Edison's introduction of utility services for electricity that created an economic boom that led to recorded music, modern movies, consumer electronics and even Silicon Valley.

Electricity is an essential component of these industries and it had to be provided as a standard component before they could flourish.

Now the internet commoditises the means of mass communication i.e. mass communication existed beforehand but the internet turned it into a standard component. Hence we've seen an explosion of industrial creativity based upon this component i.e. Google, Facebook, Twitter etc.

Each time an activity - whether electricity or mass communication or trade - is commoditised, we see explosions of creativity, the formation of future industry and the disruption of past industry. As Edison commoditised electricity provision, new industries formed and past industries such as Gas lighting companies were disrupted.

Those past industries had a choice. They could either consolidate, acquire and adapt which is what they did or they could have tried to get Congress to pass legislation to stifle the electricity market. Had those Gas Lighting Companies succeeded in doing that, then Edison and those future industries such as Hollywood, Silicon Valley, General Electric would never have formed in the US. Instead they would have formed somewhere else and the US would be a fraction of the economic power that it is today.

SOPA is simply an attempt by past industries who face disruption AND refuse to adapt, to persuade Congress to legislate in favour of past models. Its effect will be the same as Gas Lighting companies persuading Congress to legislate against electricity. It's an economic blunder.

You live in global economic market and you compete against other nations. If because of concerns over piracy the US makes such an economic blunder, then as competing nations we will act like pirates by plundering your future. If we don't, China will.

We will happily take Silicon Valley off your hands. Send it to London, we would love it.

We will happily have those jobs, those future industries. If you don't want it, we do.

They key point to understand is the internet is an essential component for future industry just like electricity. Mess with that at your peril.

Of course, Media industries complain about a changing world, they've been crying wolf for decades : "8 track tapes and piracy will destroy recorded music", "video and piracy will destroy the film industry", "internet and piracy will destroy …" blah blah blah blah blah.

Adapt or die is all I'm going to say and if you don't want those future industries please send them to us where we would care for them. So as Alien from a competing nation, I'm all for Congress destroying the US economic future. Go for it.

As they say, fortune favours the brave or more aptly :-

"OUR FORTUNE IS FAVOURED BY CONGRESS CHARGING BLINDLY INTO AN ECONOMIC ABYSS"

We love you Congress.

PS if you could also persuade the Murdoch empire to shift permanently to the US that would be cool too. We've been giving them hints in the UK but I'm not sure they've got the message.

Review of Mystic Me 4.0

2012-01-06T12:32:00.002Z

Before giving predictions for 2012, I'd better start by reviewing last years.

Of the 51 separate component predictions made (grouped into ten categories, see below) then :-

1 is yet undecided
4 are clearly incorrect
46 are demonstrable

This gives a 92% rate of accuracy for individual components. I'll publish the data when I get a spare moment and have completed the next round of predictions.

However, spotting individual components and trends is the easy bit (i.e. big data will become a hot topic etc). Combining it all together into a coherent story is the real trick.

When scoring a category, every single component prediction in that category must be accurate, timely, spot on etc for the story to be considered accurate. In other words, if a tiny part of the entire prediction for that category is wrong then the whole thing is wrong – no excuses.

I don't believe in the idea of "well I was 70% correct in my statement", it's simple binary - yes or no.

The results are as follows :-

#	Category	Notes	Score (0-1)
1	Cloud	All ten components are demonstrably correct	1
2	Environment	Of three components, two are demonstrable, one has yet to complete but can be considered likely to fail.	0
3	Economy	Of nine components, eight are demonstrable but one is clearly wrong (i.e. the FTSE did not drop below 3,000).	0
4	Society	All four components were demonstrably correct	1
5	Technology Business	Of seven components, two are not demonstrable (i.e. VMWare will increasingly act as two operational divisions and CPTN holdings will turn out to be a patent troll)	0
6	Media Technology	Of seven components, one was clearly wrong (i.e. government regulation to introduce censorship based services designed to "protect the most vulnerable")	0
7	Manufacturing Business	The one component prediction specified is demonstrable	1
8	Words to watch for	All five component predictions are demonstrable	1
9	Social Mobility	Both component predictions are demonstrable	1
10	MISOG's	All three component predictions are demonstrable	1

Overall this gives 60% accuracy but since the goal was to increase specificity to achieve a target of 50% accuracy then I can conclude that I overshot the prediction target and 2012's predictions will have to become even more demanding and more specific.

Hence the result is close but no cigar

For reference, the list of component predictions were:-

#	Category	Component Prediction for 2011
1	Cloud	Conventional wisdom within the popular press shifts towards seeing open source architectures dominating the cloud computing space
2	Cloud	Cost efficiency arguments around cloud computing will increasingly be replaced with customer innovation stories
3	Cloud	the adoption rates of cloud computing will outstrip many early analyst predictions
4	Cloud	Pundits will cite AWS as exceeding $1 billion in revenue
5	Cloud	Enterprise IT will increasingly focus on new value creation, architecture and vendor management techniques
6	Cloud	Increasing mention of terms like supply chain management and new business models based upon outcome
7	Cloud	Platform as a service (PaaS) will overtake Infrastructure as a service (IaaS) as the main buzz of cloud computing
8	Cloud	There will also be no let up in the pace of mergers and acquisitions in this industry
9	Cloud	Governments will also increasingly become engaged in discussing regulation of the cloud
10	Cloud	Some official will be talking up the idea of licensed cloud operators
11	Environment	Total Arctic Ice volume will decline to the lowest level on record
12	Environment	The melting season is considered to have extended by several weeks
13	Environment	The UK will suffer another cold winter.
14	Economy	Inflation, as measured by RPI, will continue to rise.
15	Economy	Because of instabilities in the recovery the MPC will hold interest rates low
16	Economy	BoE will implement a last gasp round of quantitative easing
17	Economy	London will experience a property bubble for high value residential property
18	Economy	The overall housing market, according to the Halifax House Price Index, will suffer a fall in prices
19	Economy	UK will fall back into recession
20	Economy	Instabilities will be driven by overexposure of banks to instruments based on sovereign debt
21	Economy	There will be increasing market attacks on sovereign debt and a drop in consumer confidence
22	Economy	The FTSE 100 will drop below 3,000 during the year.
23	Society	We will see increasing civil disobedience in many countries.
24	Society	UK will experience increasing protests and strike action
25	Society	Despite the necessity to reduce debt, the coalition (in particular the Liberal Party) will continue to wain in popularity polls
26	Society	Despite pundits predicting collapse of the coalition, it will muddle through.
27	Technology Business	VMWare will increasingly act as two operational divisions - one focused on infrastructure, the other on platform. Some public pundits will start to question whether one of the units will be sold.
28	Technology Business	CPTN holdings will turn out to be a patent troll, its target is not Android or FOSS specifically but Cloud in general.
29	Technology Business	We should see examples of companies trading on variability in cloud infrastructure prices through the provision of true brokerage services
30	Technology Business	The volume of tablet sales will sky rocket with new competitors flooding into the market
31	Technology Business	Existing industry will see a decline of traditional laptops
32	Technology Business	The concept of social searching will become increasingly important with a continuation of the plethora of start-ups providing new ways of ranking, mining and determining social reputation
33	Technology Business	Pundits discounting the future of Google will get a rude awakening
34	Media Technology	In the UK, there will be further high profiled efforts to carve up the Internet.
35	Media Technology	The use of government regulation to introduce censorship based services designed to "protect the most vulnerable"
36	Media Technology	Paywalls will continue to be the rage
37	Media Technology	Largest effect will come through the proliferation of devices with on-chip DRM
38	Media Technology	Media pundits will raise the question whether these devices and the introduction of two tier environments means the Internet can be effectively controlled for the average consumer
39	Media Technology	Online video will continue to grow exponentially
40	Media Technology	YouTube becoming increasingly seen as the future distribution channel of media
41	Manufacturing Business	Printed electronics will have a robust year in the popular press, with pundits talking up the potential for this technology especially when combined with 3D printing
42	Words to watch for:	Consumerization
43	Words to watch for:	Shadow IT
44	Words to watch for:	Ecosystem
45	Words to watch for:	Cloud computing will still cause confusion
46	Words to watch for:	There will be a continuation of marketing efforts to distinguish between enterprise and public cloud
47	Social Mobility	Increases in tax and a crackdown on tax avoidance
48	Social Mobility	No mass exodus of wealth from the UK.
49	MISOG's	There will be a considerable amount of grumbling over the Royal Wedding and how much coverage it's getting.
50	MISOG's	Unfortunately no private company will step upto the plate and offer to pay the bill
51	MISOG's	Someone, somewhere will write an article about how the cost of giving everyone an extra days holiday could stall the UK recovery

Future costs and Cloud

2011-12-07T20:03:00.015Z

There are many subjects which I find tiresome but two which are starting to irritate me are the notion of Enterprise Cloud and Financial ERP. I'll deal with Enterprise Cloud in this post.

The shift from products to utility services inevitably incurs various forms of risks. These include disruption risks such as loss previous skillsets and political capital to transitional risks such as changes to governance and transparency of suppliers to outsourcing risks such as pricing competition and loss of strategic control.

A common, past method of dealing with transitional risks is the use of a hybrid model combining both public and private supply. However, this is a transitional approach and should be undertaken with a view of moving to a future hybrid model of multiple public providers (i.e. a competitive market).

A transitional approach requires you to build in a way which is likely to be compatible with a future public market. For infrastructure this mean use of commodity components and in most scenarios an EC2 / S3 / EBS like interface. Hence my support for efforts like OpenStack (and to a lesser extent Eucalyptus).

Unfortunately, many applications are designed with the best practice for a product world i.e. scaling is about bigger machines, resilience is about N+1 and in general the focus is on reliable hardware. Best practice for a utility world involves resilience, scaling and failure modes built around software i.e. design for failure, distributed systems and chaos engines such as Netflix's chaos monkey approach. There is an inevitable cost of architectural transition from one set of best practices to another.

Obviously many companies don't like this architectural cost of change and hence want to minimise it which has given rise to the concept of the Enterprise cloud i.e. it's like cloud but without the commodity bit.

It should be noted that cloud is simply a result of a standard process of evolution that inevitably leads to operational efficiency through provision of a commodity. A consequence of this is it also enables higher rates of innovation for new business activities (such as big data) through the combined effects of componentisation and creative destruction. The upshot of this, is that you've never had a choice with cloud - it's just a question of when and the longer you leave it then the more you put yourself at a competitive disadvantage to others.

Whilst a commodity based private cloud (which can and should achieve much lower costs than public provision today) is a viable option in the short to mid term (depending upon scale), unfortunately Enterprise clouds don't move you along that architectural transition and here there's a real gotch'a. The problem is simply known as Jevons' paradox.

As competitors gain the benefits of more efficient commodity provision and higher rates of creation, this is unlikely to result in a reduction in IT budgets but instead more IT activities undertaken with everyone trying to keep up with each other. We've seen this for the last thirty years i.e. as IT has become more efficient, IT budgets haven't fallen but we've just ended up doing more stuff.

You therefore have to factor in that six years from now your architectural transition costs may well have spiralled by an order of magnitude due simply to the increased size of the estate. This can obviously be counter balanced with a simplification strategy, assuming your estate is already bloated but when considering Enterprise cloud, you must include this increase of architectural transition costs along with less efficient provision during that time due to a non commodity approach.

In most cases, Enterprise cloud will be a pretty unattractive option and you should be very careful before embarking on such a route. It can however still be useful as part of a sweat and dump strategy for legacy environments i.e. you push the capital costs for legacy onto a provider with a view of dumping that part of the estate in the near term.

Why do I find this subject irksome? I simply hate repeating old ground and this has been covered many times before over many years. This is the last time.

Spotting the Next generation ...

2011-11-21T15:53:00.010Z

For the best part of a decade I've worked on how business activities evolve from genesis through to products to commodity. As activities evolve then practices (such as management methods) that are applied also have to change leading to one size never fits all.

I've also demonstrated how this process of evolution creates a cycle, with commoditisation enabling further genesis (as in innovation of new activities) and the importance of inertia barriers in regulating this cycle.

These cycles also appear at the macro-economic scale and they're known as Kondratiev waves, however even more localised cycles (i.e. specific to an industry) tend to be associated with organisational change.

Hence with the commoditisation of the means of mass communication (i.e. the internet), new forms of organisation appeared known as Web 2.0. These companies had different activities, practices, strategies and structure to the more traditional, enterprise organisations that existed before.

In general terms, this cycle of evolution and its interaction with inertia barriers creates three economic phases - peace, war and build. During the peace phase, sustaining change is particularly high, companies fight to maintain their relative position and inertia barriers to change help support the status quo.

The war phase occurs when an activity evolves from one state (such as products or rental services) to another (such as commodity or utility services) and during this time disruptive change dominates whilst inertia barriers become a hindrance to survival. Companies literally are fighting to survive, often against their own internal culture and inertia to change.

So during the last IT war, caused by the evolution of the means of mass communication, these new web 2.0 organisations formed and the rate of disruptive change was high. Those traditional, more enterprise like companies that were impacted but survived the war had adapted and learnt many of the lessons of web 2.0. Today, social networking, social media, rich internet applications are the norm.

However, cloud computing - which is nothing more than the evolution of a wide range of activities from products (and rental services) to commodity (and utility services) - has initiated a new phase of war.

There are currently three main forms of IT organisation in the wild - the traditional enterprise who have adapted, the web 2.0 players and a new breed of Next generation companies. The practices, activities, strategies and organisational memes of these Next generation are the ones which will tend to diffuse and dominate our industry.

The underlying framework, models and principles of how this works is part of a report I'm finishing for the LEF. Including in-depth surveys of over a hundred companies, it provides a reasonable hypothesis of organisation evolution across activities, practices and strategies. I say reasonable because it's more than postulation as there is data that demonstrates the pattern but this is not peer reviewed, the sample sizes are small and hence its only a reasonable hypothesis and requires a good dose of skepticism.

I'm not going to go through the report but I thought it might be useful to just comment on how to identify the Next generation in general terms.

If your organisation :-

embraces change and is acutely aware of inertia barriers
has primary goals based upon disruption of existing markets and growing ecosystems rather than a focus on profitability and growth into emerging geographical markets
views commoditisation not as a liability but an opportunity
operates through small, highly empowered teams building and creating services
builds and exploits ecosystems by enabling others to build upon its services
uses data analytics rather than metrics to determine action
continually challenges and removes processes
understands evolution and avoids one size fits all mentality
builds distributed systems designed for failure that use commodity IT where possible
considers itself to be as, if not more, efficient than Amazon
games behaviour both internally and externally
views culture as malleable and focuses on talent density
views open source as a tactical weapon
is seen by others to be simultaneously highly innovative, customer focused and efficient despite what Porter said on the subject

... then you're Next generation. If this isn't you, you're not. If you're not, then just hope this war doesn't stretch into your industry or that IT isn't a major barrier to entry into your space or this hypothesis is completely wrong.

Along with retail, media, travel firms and others who have already suffered in the last war - we should see software products & hosting, insurance and retail banking dragged firmly into this one.

Of course, along with this war phase comes a build phase, a golden age where the genesis of new activities flourishes and we seen periods of frenzied growth.

The cycle is continuous, and the next major war after this one should be related to the commoditisation of the manufacturing process itself ... now, that'll be a big one. For me, this is personally important as it creates an opportunity to test the hypothesis.

What we should see as the means of manufacturing becomes more of a commodity (driven by 3D printing, printed electronics and hybrid printers) is a high degree of disruption in manufacturing industries as inertia barriers to change are broken by new entrants whose practices will be remarkably different from the existing players. The practices of these Next Next generation of company will diffuse and become the norm.

If this hold true, then I'll have enough to finally put the work up for peer review - having initial data (covering 4,000 data points), a hypothesis which describes a pattern from that initial data and has been preliminarily shown to exist in this current war through a survey (providing in total over 6,000 data points across 100 companies) and finally a pattern which is predictive.

If it doesn't and is falsified then it's back to the drawing board. The downside is I'll have to wait many years (possibly as much as a decade) before the next war starts ... if only companies were more like bacteria that I could grow in a lab.

Bye, Bye American dream ...

2011-11-05T12:21:00.017Z

In the last fifty years wealth inequality has risen dramatically in the US, with real earnings (adjusting for inflation) increasing by 5% at the lower end of the scale and by 700% at the upper end of the wealth bracket. Not that inequality itself is a negative thing, it can spur ambition.

Unfortunately there is a strong correlation and causation between wealth inequality and intergenerational social mobility. Hence around the 1950s-60s, social mobility was around 12% in the US i.e. 1 in 8 of people in that era moved up (and down) a social class. If you were born into the working class around the 1950s-60s then you almost certainly know several people who have made it big, who have achieved the American dream, maybe you're one. Today, social mobility languishes around 3% i.e. 1 in 33.

Chances are that people born today in social working classes will see few of their contemporaries (if any) breakthrough to higher social classes.

In the US today, your future social class is not predominantly defined by your ability but by who your parents are and the opportunities they can provide you. As this spreads, the lack of role models for a change in class should become self reinforcing.

In short, if you're born today to a wealthy family, hey presto ... you're a success ... but if you're born today to a family of police officers (a noble profession in my view) chances of you becoming a CEO of a major corporation are pretty slim, certainly much less likely than it was for those born in the 50s or 60s.

I mention this because a friend made a comment that they thought that the concept of monarchy and class was over. Certainly in the UK we've suffered with poor social mobility and its impact on competition but even our social mobility is now better than the US. If anything, hereditary monarchy is alive, growing and becoming stronger in the US through hereditary financial monarchy. All Hail, the new Kings and Queens.

As for the American Dream, to be brutally honest the last vestiges of this are standing outside Wall Street in a much derided occupy movement. Certainly there will be examples of people who buck the trend (i.e. 1 in 33 or less) and there is always the X-Factor or adoption by a wealthy family.

My only advice is that if you're born into a low social class in the US, blame your parents and learn to curtsey or doth your cap. If your're born into the high social classes, thank your parents, lord it up but watch those peasants - especially when they get hold of pitchforks.

Bye, Bye American Dream ... nice idea, shame it didn't last.

[As an addition : for those who want to learn more on this subject, a good starting point is probably the Applied Economics 2009 paper from Harvard University scholar Dr Dan Andrews on "More inequality, less social mobility"]

Is it time to reset?

2011-11-04T11:02:00.020Z

Our economy faces unusual challenges due to the exceptionally high debt burden. In terms of Government debt this is represented through Gilts (Government backed securities of debt). The recommended solutions to our problems usually involved manipulation of the money supply in the hopes that somehow this will correct the underlying problems.

For example, in the UK the Government sells debt with Gilts then, through a wheeze known as Quantitative Easing, it prints money to buy back the Gilts it has sold. The net effect is that GBP is devalued (leading to commodity price inflation), the delta between buy and sell price is absorbed as profits by the counter party (normally a bank) and the entire distribution of wealth becomes further concentrated to the advantage of the counter party.

It would in effect be more cost effective (and honest) if Government's didn't sell and buy back Gilts but simply printed money to counter their shortfall. You'd still however be left with the issue of devaluation.

Now in an export led economy devaluation can help boost exports at the same time as increasing inflation. However, we're not an export led economy, so devaluation without direct investment leads to higher material costs, less profitability in internal markets and correspondingly cost reduction, weakening of the internal market, higher inflation and stagnation i.e. what the media call stagflation.

All of these problems are exacerbated by debt, inequality of wealth distribution etc. In the Eurozone we're seeing this shake out with Greece which faces either a decade or more of misery or default on the Government debt (i.e. gilts)

Now, a Greece default would be a disaster because it would be difficult to raise further debt and investors would be wary of trading with Greece. But wait, it's part of Europe ... the EU could always raise debt for it. I suspect this is why people are so keen for Greece to leave the Euro if it defaults. A country that defaults but then continues is not a message the market wants to hear.

Why?

Well despite media opinion, Government shouldn't operate for the benefit of investors but society as a whole and Gilts come with risk (albiet relatively small). Now if a country default and continued to operate, then the problem for the market becomes what if this idea extends? So let us extend this Greek idea a bit more. What if every country in Europe and ideally add in the US as well, simultaneously defaults on Government Debt i.e. Gilts and resets the debt to zero?

Well for investors it would cause one hell of a haircut and some banks would probably fail. However, those can be nationalised where necessary and measures taken to limit the damage. You'd certainly need some form of protective measures for those at the bottom end of society - hence tax raises would be necessary - and Government would need to co-operate to smooth out some of the fall-out.

There'd be lots of nashing of teeth, the markets would suffer turmoil whilst they reset but investment (in the case of shares, gilts etc) is always shorter term gambling and despite the losses the size of the EU & US means investment will still continue, possibly even flourish. It's worth remembering that market indices (FTSE etc) are also not a good indicator of the underlying economy despite the media's fixation on them.

I can't see there are actually any real downsides to a Europe and US default (bar the temporary turmoil etc) except for investors but such gambling inherently occurs risk and these are fairly unique circumstances. The Governments could always shore up the economy by direct Keynesian style investment as well.

Naturally, some would argue that this might collapse the USD / EURO / GBP against other currencies such as the Yuan. But the above approach wouldn't significantly increase money supply (other than investors dumping) and the Yuan isn't a free floating currency. Certainly against other countries they'd be possible devaluation but then given the size of the trading blocks the effect should be minimal over time. What about sanctions? Against Europe & the US - don't make me laugh.

Of course, some will argue it would spook investors so much that there'd be a collapse of investment. However, if the money was invested in Government debt it wasn't invested in longer term infrastructural goals which is what we actually need. Any investors sent to the wall would be quickly replaced by others - that's capitalism for you - and as for pension funds, well it's true that we'd need decent protective measures put in place to protect the poorest but beyond that not a great deal more.

I'm writing this because the UK Government is planning to give more taxpayers' money to the IMF who may contribute to the Eurozone bail-out (something which our Gov. said they wouldn't do) or in other words piling on more debt to help solve problems caused by having too much debt.

This is a cycle we need to break. Can I suggest we reset the market economy by resetting Government debt. A simultaneous default by Europe and US on all Government debt should do the trick.

Does this post mean I believe this is the right course of action? Well, unfortunately whilst the idea is interesting, the practice is virtually impossible. In order to exact such a change, the various Governments would need to agree and execute almost instantly to avoid investors dumping Government bonds on the unsuspecting public (or wrapping them up in some other complex instrument). I have doubts that our leaders are capable of such consolidated action. All it would take is one party to phone a friend and warn them to get out of Government bonds and the entire scheme would unravel quickly.

So, overall ... I agree with the concept but the practice is probably too difficult and it's much more likely that we'll settle for many years (or decades in the case of Greece) of austerity

Ecosystem & Porter

2011-11-03T13:52:00.007Z

A common characteristic of Next generation organisations (as opposed to traditional) is their focus on ecosystems and the provision of platforms to support their growth. The purpose of such ecosystems is not simply some form of marketing exercise but instead a mechanism for managing the innovation paradox (i.e. the need to be efficient to compete today but also to be creative in order to compete tomorrow).

Consider the provision through an online API of a software system whether it’s SalesForce, Amazon’s AWS or PayPal’s x.commerce platform. These services are core utilities that the organisation is providing with the express aim of others consuming i.e. an ecosystem of consumers developing around the service.

The consumption of the service may represent general use or even novel and more creative uses e.g. the early provision of big data Hadoop systems on AWS. Since the genesis of any activity is uncertain (being chaotic) and likely to fail, then the use of utility services helps reduce the cost of failure and thereby encourages the creative pursuits of others. The larger the ecosystem, the more likely that creative models of consumption and the genesis of new activity will be occurring.

Naturally, if those activities are useful (i.e. Hadoop on AWS) they will spread through the normal process of diffusion. By monitoring the ecosystem’s use of your services this spread can be detected.

These factors enable a model known as ILC (innovate-leverage-commoditise) to be used (see figure below). Through provision of utility services and the development of an ecosystem, you enable others to create new activities at a lower cost of failure and hence encourage innovation to occur in the ecosystem and around your services. Through monitoring, you can leverage the ecosystem to identify novel activities that are spreading. An organisation then can either copy or acquire such activities and commoditise these to create further services that enable growth of the ecosystem and hence further innovation through componentisation effects.

For example, the introduction of AWS enabled others to build Hadoop on AWS. With the spread of Hadoop on AWS, Amazon introduced an equivalent utility service - Elastic Map Reduce (EMR) - which in turn has enabled novel activities to appear that consume EMR. And so the cycle repeats ...

Figure 1 - ILC model (click on image for higher resolution)

NB, I italicise the term innovation because I'm referring to the creation of novel activities i.e. genesis of an activity rather than the broad use of the word innovation which is applied to almost everything from feature differentiation to service provision of a pre-existing activity. See "The Abuse of Innovation"

It’s through models such as ILC that an organisation can simultaneously appear to be :-

highly creative - by pushing such uncertain activities to a wider ecosystem
customer focused - by leveraging the ecosystem to identify that which is becoming adopted
highly efficient - by focusing on commoditisation

In Porter’s terms these Next generation organisations have a primary focus on a cost leadership (a best price) for provision of the utility service.

BUT they also have a strong differentiation strategy which is heavily influenced through creative pursuits of others within the ecosystem that develops around their utility service (i.e. genesis of novel activities being driven outside the organisation). This is why Next generation organisations often cite “enabling others to build upon our services” as criticial.

BUT they also have a strong customer focus strategy heavy influenced by adoption within the ecosystem (i.e. they leverage the ecosystem to identify activities that are spreading) and subsequent provision of these activities as further utility services.

In such cases, all three of Porter's strategies are being pursued simultaneously with the major nuance between these players is whether they use copying (a weak ecosystem play) or acquisition (a strong, reinforcing ecosystem play).

I mention this because "focus on customer, innovation or efficiency" is one of those truisms like "culture eats strategy for breakfast". It's a great sound bite but on closer examination, it doesn't seem to stand up to rigorous scrutiny.

I'm currently collecting a selection of truisms and each one of them seems to leak like a sieve when exposed to rigorous study. The following are ones which in my view are all in need of serious re-evaluation :-

You can't manage what you can’t measure
You need to give customers what they want
The best way to predict the future is to create it
Avoid the commoditisation trap
Business has only two functions - marketing and innovation
Culture eats strategy for breakfast
Focus on customer, innovation or efficiency

More QE madness ...

2011-10-08T19:19:00.011+01:00

In the last five years, GBP has dropped in value by between 23-47% against the USD, EUR and CNY when Gold is used as the standard. GBP has taken a pasting. Our UK inflation during that time has been around 18% (based on CPI) but that's a basket of goods approach, as we've all seen basic commodity inflation has been much higher. You can pretty much put the entire inflation down to GBP devaluation.

One might argue that devaluing GBP is good for exports, however the UK has a huge balance of trade deficit managing to clock up a record breaking £9.2 bn in Dec'10. Whilst exporters have raised some recent cheer, what has to be remembered is that the UK imports almost all its copper, ferrous metals, lead, zinc, rubber, and raw cotton and about one-third of its food, along with a whole range of finished goods. With the GBP down the toilet, the cost of these goods has risen dramatically and certain items just aren't luxuries.

So, who has made good with the collapse of the GBP and where did it come from? Well, the reason why GBP has been hit hard is basically because of monetarist policies such as quantitative easing and keeping interest rates low. Quantitative easing (QE) is the best wheeze of the two and consists basically of buying back debt we're constantly selling, at an obvious premium, and hence simply amounts to printing money and handing it over to banks. The monetarist delusion is that those banks will lend it out, increasing money supply, and hence magically all our problems will be solved. In reality it takes no consideration of capitalisation requirements, debt exposure, weakness of infrastructure, business cycle etc. The net effect is always bad for GBP.

Through financial engineering we've created a situation of increasing inflation (due to GBP devaluation through money supply manipulation) combined with a weakness in the economy as a whole since we're not bothering to directly investing in it and trying to combat the economic cycle. This combination gives us stagflation = stagnation of industry + inflation.

So who benefits? Well, a depressed GBP makes the stock market and housing more attractive to foreign investors. Hence the FTSE, though fluctuating, remains artificially high and London housing is undergoing a mini-boom. Hence some people in the city think we're doing well, when in reality they might be doing ok but the rest of country is sinking fast.

Of course, viable solutions to this problem which include increasing interest rates, direct investment in industry (rather than QE) etc will cause lots of short term pain for those with debts and exposure combined with haircuts for banks and investors. Rather than face this, we continue on a path of devaluing our future in the belief that somehow this can correct problems caused by borrowing too much from the future. It's like a drunk reasoning that the way to deal with their drinking debts and find the path to happiness is through selling a kidney and only buying methylated spirits.

So, does our current policy help us? Not a jot. The increasing focus on money supply will probably mean more QE. At the same time, the overall economy will continue to weaken, unemployment rise, tax receipts reduce creating a need for more cost cutting, cost of basic goods will continue to increase, export led business (without direct investment) will continue to flounder but at least the FTSE will look somewhat rosy until reality catches up with it and graphically demonstrates the error of its ways.

Eventually, if we keep on this route we will finally reach a tipping point and enter the world of hyper inflation combined with extensive contraction of industry. I can't think of a nifty combination of these terms, so I'll resort to, Hyper inflation + Contraction = "Zimbabwe" Economics. Of course, I use the phrase loosely here because to actually achieve the stunning results of Zimbabwe requires a very rare kind of recklessness (or more appropriately "wrecklessness") which is unlikely to be repeated.

I'm writing this so as we enter these more troubled times, can we please, please, please remember to round up all the monetarists and either lock them up in the Tower of London or ask them to provide the data which demonstrates why what they're doing makes sense.

Larry offers Hotel California ...

2011-10-06T12:41:00.012+01:00

According to CW, Larry recently raged over Salesforce describing it as a roach motel and pleading the case for interoperability. He's just given a gift horse to one of the smartest operators in town and this time Larry's forgotten to fill it with any of his own soldiers.

First, some background which everyone knows already, so I'll keep it short :-

Activities evolve and our industry has been shifting from a product to a utility service world. This has been clear for the last 6+ years, Salesforce knows this and they've been positioning themselves in that future space.

Past success always acts as an inhibitor to future survival, it creates an inertia barrier to change. This is why Amazon and not some hosting company encumbered by an existing business model made the break into IaaS. This has been crystal clear for 4+ years. Salesforce knows this, it's why Oracle has been slow to react to the change.

In this future world, competitive markets will become key to solving those outsourcing risks such as pricing competition, second sourcing options and loss of strategic control. Such markets will require multiple providers, access to code and data (i.e. standard data formats and APIs) and semantic interoperability. The latter point is only solvable with complex systems through running code and unless the market intends to be a captured markets (i.e. dependent upon one vendor) then that code will have to be open source. This has been clear for the last 5+ years. Everyone knows this just a lot of people refuse to believe it usually because of inertia barriers which have become institutionalised.

Critical in this new world is the development of ecosystems as these enable a company to solve the innovation paradox and simultaneously appear more innovative and highly efficient. This has been blindingly obvious for 3+ years. More details on common models such as ILC can be found here. Salesforce knows this, they've been playing an acquisition game around their own ecosystem and sending market signals because of this.

With a large enough ecosystem, you can create network effects through aggregated data e.g. market reports. This can be used as a soft form of lock-in i.e. even if you open source an entire system, your service still maintains an advantage simply because of the number of people using it. In other words, you can be entirely open but in effect create lock-in (i.e. gravity) for your service because of the benefits that being within that ecosystem brings. This has been painfully obvious for the last 3+ years.

Salesforce has also been playing a tower and moat ploy, building a tower of core revenue surrounded by a moat of high barriers to entry and devoid of differential value. Attacking Salesforce is a tough call for anyone, hence I suspect Larry's aim to make interoperability his calling card.

Salesforce has the ecosystem to play an aggregated data game i.e. free market reports for an industry based upon aggregated data or free comparison KPIs to your sales team effectiveness etc. Given the smart plays Salesforce has been making, you can bet your bottom dollar they've got lots of this in the pipeline.

Salesforce could also use open source as a tactical weapon in this space. They could open source the entire system and say "come and compete", "run it yourself" with full knowledge that those who build it for themselves and take the private road will eventually switch to public, whilst those setting up as public providers will lack the ecosystem and hence any aggregated data benefits. Salesforce is also smart enough to know that this game could be played against them, so they'll have to go down that route at some point. Hence you can pretty much bet your bottom dollar they've been working on this.

Larry has walked into a huge trap. He's just called out interoperability as the key differentiator for his service but as we all know the real issue is portability which requires semantic interoperability and running code. All Salesforce has to do is start launching more aggregated data services and open source the entire system under a banners of "Freedom in the cloud", "Run it yourself for Free" and Larry is left standing with the high cost proprietary service with no real portability (except between one licensed version of Oracle and another).

It's difficult to see how Oracle's strategists could have been more tweedledum or tweedledee as currently they are primed to become the industry's example of Hotel California (you can go anywhere you like as long as you're paying fees to Oracle?).

Now, open sourcing won't be easy for SFDC because they have an existing service, security professionals will be concerned over exposing security weaknesses, lawyers will have their usual collywobbles over IP and financial controllers will gasp at writing down a technology asset.

However Benioff like Maritz (you don't think CloudFoundry doesn't have a grand strategic purpose do you?) is a shrewd player. It all boils down to a question of timing but we should be expecting checkmate to Salesforce in the near future.

Bad move Larry ... really bad

Bank Recapitalisation ... Pirate Style

2011-10-05T12:43:00.005+01:00

After much ado, the European Union seems to have been badgered into emergency action in order to re-capitalise the banking system due to its over exposure to instruments based on sovereign debt and the reliance on the dollar. I'm not a fan of this, this is just another monetarists prayer to the altar of "no sodding evidence whatsoever" and as usual the taxpayer will foot the bill.

Naturally, there will be wormtongues who will claim it was a Keynesian approach when it all goes spectacular wrong ... that is par for the course for economic banter.

So given that we're going to re-capitalise the banks, let us at least try and arrange the situation in the interest of the taxpayer. First, re-capitalisation should be forced and not voluntary and the capital ratio set by Basel should be raised to 30%. Next the banks should be given two options - either raise the money yourself or borrow from us, the nice friendly EU.

Of course, being the lender of last resort, there will be a couple of strings attached to the capital we lend (oh and lend is the operative word). Hence :-

The entire capital lent will need to be repaid annually over 5 yrs at EU base interest rates or average EU inflation (whichever is higher) + 10%.
The EU takes precedence over all other debtors and the entire banks assets will be used to underwrite the loan
A sum of Bank equity equal to capital lent will be paid to EU as our "setting up administration fee"
Late payment will incur an APR of 200% plus a penalty of 50% on any remaining capital.
No dividends will be paid until the entire loan is repaid
Upon final repayment of the loan, another sum of Bank equity equal to the capital + interest + any late payment fees will be paid over to the EU as our "closing administration free"
If you don't like the terms then go raise the capital yourself on the open market.

Now, I'm not actually advocating such draconian terms but I'm arguing for the EU and our Gov to stop acting like they're just pawns in this global game and start acting like pirates. The banking system is an essential vehicle for our economic system but like all things, it should be managed in the wider interest of society.

Why I believe AAPL will crumble ...

2011-10-04T18:46:00.007+01:00

Earlier this year, at the height of Apple fever, I made a bet that Apple will be in Chapter 11 by the end of 2017. I thought I'd explain my reasoning because it's not what most people would suspect.

First, the problem with Apple in my view was Jobs. Whilst Steve Jobs was outstanding at creative leadership, that is only part of the battle for creating a sustainable company. An exclusive focus on creative leadership always leads to failure as the genesis of new activities (i.e. innovation) might be high worth but it's unstable and uncertain.

The problem for Apple started in my view from a major strategic blunder - it didn't open source iOS, it didn't feel it needed to, it was building the entire stack. By not doing so it enabled Android to exist. Apple gave oxygen to the formation of a competitive ecosystem of hardware providers to develop around Google's new weapon.

As that ecosystem develops, Apple will find itself in a stand alone innovation game against it. The pressure will build for ever more outstanding and exciting breakthroughs in technology which Apple has delivered with the iPad. Unfortunately, this pressure will continue and such breakthroughs by their very nature (chaotic) are uncertain and every company in this position before has failed.

Take Commodore and the Commodore64 which was vastly more influential than the iPad. The C64 transformed a world where computers were rooms owned by huge corporations into personal computers. In terms of consumerization, the C64 was dramatic.

Commodore and Apple both tried to lead this new world through constant innovation but were hammered by the more commodity based ecosystem approach of "IBM PC compatible". Commodore died and Apple barely survived but unfortunately it seems to have failed to learn that lesson.

So once again, we find ourselves in a world where Apple is pushed into the high risk stand alone innovation game against a growing, more commodity focused, ecosystem. That ecosystem will enable rapid innovation of higher order systems, it will outstrip Apple once again and I suspect that Cook (the new Apple CEO) knows this.

The only viable defence against such an ecosystem play is to build a bigger ecosystem (which is tough as a stand-alone) or to buy up the supply chain and use patents to slow your competitors. The latter Apple has done but such moves only slow the change, they don't stop it.

It can give you breathing space though to find that next breakthrough or to work out how to build a bigger ecosystem. However, the problem is often expectation i.e. your customer expect that breakthrough continuously.

I've not listened to Apple's latest press release but if its lacks any breakthroughs and dazzling tech (which I strongly suspect) then markets and fans will slowly turn against Cook and cry "bring Jobs back". Markets always do this, they always want more of the past.

Into this current fray, Amazon will certainly push with its normal approach of commoditising an industry and building an ecosystem around itself. If Google and the greater ecosystem around Android have been waiting for this moment, then they'll shortly strike at Apple - a flood of patent attacks.

Apple will start to turn inwards and the market pressure on Cook will intensify. They'll go from looking for that next breakthrough to needing it. Culture will start to change, it may start to buckle.

Apple's core business will be undermined by the commodity players whose technology will rapidly catch up and overtake, assuming Google can get them to work in a common interest. Soon Android devices will be everywhere.

If Apple's patent and supply chain protection measures fail, if a concerted patent attack against Apple is successful then this will happen sooner.

At this point, with margins under pressure, markets under attack, the gloss peeling off the Apple logo and the culture starting to decline then the markets will go after Cook - "it was his fault" they'll say.

Of course it wasn't Cook's fault, Jobs made the blunder with an excessive focus on creative leadership creating a high margin but unsustainable business.

Cook might pull out a miracle and maybe they've got some tech they've been keeping back in preparation to dazzle. Maybe he'll help Apple create that sustainable company which balances both innovation and commodity by dealing with the constant flow between them.

I doubt it, markets never think that hard nor give that much time. Cook is more likely to end up as the next Leo Apotheker ... and as for Apple well it didn't learn the lesson first time around, I don't suspect Google and Amazon will let it have a third go.

That's my view, that's my reasoning and that's why I made my prediction. Of course, the prediction assumed Jobs would still be the CEO and maybe Cook can change things by correcting those errors. Should be interesting to find out.

Why proprietary software is mathematically more evil than open source

2011-10-04T14:25:00.008+01:00

As we all know, proprietary software costs time (t) and money (m). Hence,

Proprietary Software = m.t

However, we also know that time is money. Hence,

Proprietary Software = m^2

However money is the root of all evil. Hence,

m = (Evil)^1/2

By substitution we can show that,

Proprietary Software=((Evil)^1/2)^2 = Evil

Ok, so proprietary software is evil but is this more evil than open source? Since open source only costs time, it is therefore obvious that:-

Evilness of Proprietary Software = (Evilness of open source)^2

As a function of cost, then :-

Evilness of Proprietary Software per $= (Evilness of open source)^2/0

i.e.

Evilness of Proprietary Software per $= (Evilness of open source).infinity

If we hypothesise that open source isn't evil then :-

Evilness of Proprietary Software per $=0.infinity = 1

In other words, proprietary software is infinitely more evil than open source and even if open source isn't evil then proprietary software still absolutely is.

Next posts include ... Mathematical proof of why patent lawyers suck your soul dry and why clothing concepts in a cloak of mathematics doesn't make them right.

Culture eats strategy ... where's the data?

2011-10-01T18:16:00.010+01:00

I find irksome the management mantra that is commonly spouted of "Culture eats strategy for breakfast" because no-one ever seems to be able to justify the statement with data. I thought I'll pen a few thoughts on this.

An organisation consists of a mass of people, activities and practices combined with reserves of physical, financial, human and social capital. It's the interaction of the former three which impacts the latter either positively or negatively.

Culture results from the interaction of people with social (e.g beliefs, values, reputation) and human (e.g. skills, knowledge, myths) capital. In much the same way, the business itself can be described through the interaction of people, activities and practices with various forms of capital.

Strategy, is simply a plan of action, an intention and an aim e.g. it's the act of trying to achieve a particular goal or result. Either something has a strategy or we leave it to chance, randomness and accident.

We often talk about product strategy, marketing strategy, business strategy and organisational strategy but equally (if not in many cases more) important is cultural strategy. If you don't aim or plan to develop a particular culture, you'll end up with something by accident and that is not necessarily a good thing.

In recent years, the creation, building, "gaming" and planning of culture has become an increasingly more visible topic. Few have highlighted this trend as much as Zappos and Tony Hseih's work on delivering happiness. Be under no doubts, you can plan to build a specific culture.

Once a culture has formed it can certainly impact what business, product and marketing strategies you can effectively deploy in much the same way that past product strategies often impact future product strategies through inertia such as concerns over cannibalisation etc. In some cases, a future product strategy may require you to plan a new culture by spinning-off a group from the main corporate body.

As a rule of thumb your future strategies are impacted by today's strategies.

Whilst I can see some modicum of merit in bland arguments such as culture trumps products in certain industries, the culture eats strategy argument appears entirely misguided because you can plan to create or change a culture. Your strategy might require you to create a new group, to focus on happiness or to game the system - it doesn't have to be random or accidental.

To cut a long story short, the "Culture eats strategy" statement hypothesises that :-

unplanned, random and accidental [lacking strategy] culture eats for breakfast having a plan, intention or aim [for culture].

... I'm sorry I don't buy that, especially unless backed up by considerable amounts of data to counter examples such as Netflix and Zappos which show the opposite.

The counter hypothesis is that having a strategy for culture, organisation, business, product, marketing etc is better than not having one i.e. strategy eats all for breakfast, lunch and tea. In other words having a plan of action, aim or intention to achieve a goal is better than relying on randomness, accident and fate to do the same.

Now, the counter hypothesis would appear to be an obvious truth which is dangerous in itself. So, I'll start the process of collecting data and let's find out whether the "Culture eats Strategy" brigade have a leg to stand on. I doubt they do but then I might be pleasantly surprised.

Strata Conference

2011-09-25T09:37:00.004+01:00

The O'Reilly Strata Conference NYC has now finished and I have to say it was a blast. The standard of speakers, corridor chat and the general environment was exceptionally high. If you're interested in speaking, they've already opened up the request for proposals for the Feb'12 event, so get writing.

There was something magical about the event in NYC created by a convergence of people, technology and ideas. I haven't seen a conference with this much buzz and excitement since ETech. You can guess that I was truly impressed, it was O'Reilly at its finest and that's a tall order given the very high standard of their conferences. If you missed the conference, then you can find many videos from the event on the O'Reilly channel.

I was also fortunate enough to be asked to speak, the video of my talk is below. In my session I covered commoditisation, innovation and the role of big data by examining some of my new research into the evolution of organisations. As per normal, the title of the talk is my usual Situation Normal, Everything Must Change and the talk itself is different from any other previous example i.e. the title applies to the talk itself.

By the way, if you have enjoyed my talks and you're interested in helping out with my research then please take 20 minutes to complete my online survey, as that would be really appreciated.

OSCON 2010: "Situation Normal, Everything Must Change"

Prisoner of T5 ...

2011-09-11T17:58:00.012+01:00

On average, I find myself on a plane at least twenty times in a year. There's a lot I dislike about flying. In no particular order my pet dislikes are: flying coach, British Airways and airport queues at customs (especially NYC & SFO where it is particularly abysmal).

I don't suffer from jet lag (I use the starvation trick when needed) and I generally find airport security pretty quick and responsive. Though I'm a smoker, I've always gone outside the building to smoke, so the movement of airports to non-smoking has been no skin of my nose.

Well, that was until today.

I arrived early at Heathrow Terminal 5 (an airport I rarely use as it's mainly BA), sat down for a quick bite to eat and then decided to go out for a smoke - obviously you can't smoke in the airport.

So I went to security who said I had to be escorted through. Kidding? I have been through every one of the world's busiest airports plus a whole host of smaller airports and I've never once needed to be escorted from departures to outside the airport. I was then told by security that I had to get permission from the British Airways customer support person - damn.

Being incredibly polite I asked the lady from BA whether I could be escorted outside to have a cigarette. "No" was the reply because there was less than three hours until my flight.

So, let us be clear ... I'm currently a "captive" inside T5 departures lounge unable to leave until I get on my flight? I'm held against my will, a prisoner of BA!!! I feel a protest song coming on ...

"Free, Free, Free Simon Wardley ..."

Joking aside, my dislike of British Airways has hit an all time low.

[Addition]

Well, it's BA so what do you think happened. Yes, my flight has been delayed close to departure by just under two hours. Still not allowed out to have a smoke - a cruel and unusual punishment.

Next phase of research ...

2011-09-10T13:52:00.007+01:00

Many years ago, I produced the ubiquity vs certainty curve to describe the process of how business activities evolve. It took 4,084 data points to create the curve and more details about this topic can be found here.

Currently, I'm researching into how organisations evolve. After conducting a number of general and then specific interviews (creating a thousand data points), I've been able to create models of evolution which hopefully I'll be using in future presentations.

However, I need to collect more data to test the models and either verify or falsify them and hence I've put a general survey online : [Link to Survey]

The presentations that I give at various conferences are based upon this process of hypothesis and testing, so if you have ever found my work useful (such as my various talks at OSCON on cloud computing, see below) then I would be very grateful if you could take 10-20 mins to complete it.

Depending upon the results and validity of the models, I'm aiming to give a number of talks next year on how organisations evolve combined with techniques to exploit this. Naturally, I'll be blogging about the findings as well and the survey does allow you to provide an email in case you'd like a copy of the overall results.

[Link to Survey]

Thanks

Simon Wardley

OSCON 2010: "Situation Normal, Everything Must Change"

The battle that wasn't ...

2011-09-02T12:01:00.004+01:00

Chris Boos wrote an interesting post about a debate that @samj and I were having on twitter regarding APIs in the cloud space. I thought I'd leave my comment here as a general view on the subject.

A couple of things to point out. Twitter is not the best tool in the world to determine the exact context of a discussion because those listening aren't generally privy to the history of the discussion. Hence in this case, it may not be clear that Sam Johnston and I are in absolute agreement on the importance of open source and efforts like OpenStack in this world.

Any difference between us is on the necessity of reverse engineering APIs and co-opting as the main short term tactical play. The long term we're both totally in agreement on - open standards, open formats and open source are critical.

Our difference in views on short term tactical plays hardly constitutes a battle but is merely debate. As for being a "giant", whilst that is very flattering it doesn't coincide with my view of the world. Nevertheless, it was an excellent post by Chris and much appreciated.

Comment --

Just to clarify my view - as it currently stands any company can reverse engineer an API for reasons of interoperability. Hence when trying to make a market of providers in the IaaS space with semantic interoperability between providers, I strongly support adoption where there is clearly a dominant API.

It should be noted that such a market can have multiple open source and proprietary implementations around the API. However, running code through an open source effort is necessary to form a market place without a single (or consortium of) vendor(s) being able to force a tax on that market. In other words, providers need to have an operational means of implementing the service and compete in the market without a necessity to purchase software licenses (a tax on competition). They may choose to buy software to do so but a free market is one unencumbered by such forced taxation.

This is why I do no support MSFT Azure's effort, despite the provision of open standards because there exist no open source implementation.

This is why I did not support Google's AppEngine, despite the provision of an SDK as there existed no fully operational open source means of implementing the service.

This is why I strongly support open source efforts which reverse engineer the dominant API for reasons of interoperability e.g. open stack, eucalyptus etc.

It is also why I strongly support open source efforts which attempt to create the dominant standard in a fledgling market, such as CloudFoundry in the PaaS arena.

Once the marketplace of alternative providers is large enough and it has the dominant ecosystem then the open source effort in effect becomes the defacto standard for implementation and the API in that market. If necessary, due to abuse of position by the original provider, then the API can be differentiated away from the original provider including providing an entirely new API where applicable.

I don't find attempts to differentiate on API in a utility world where one API is clearly dominant meaningful. Of course if an open source effort (such as openstack) creates a large enough ecosystem then it is in effect the dominant and can do as it pleases.

I find re-inventing the wheel by creating an API by committee and attempting to get the market to adopt as a wasted effort when a market has in principle chosen.

I do find the way to standardise is through creating the largest ecosystem and in such cases both reverse engineering the dominant API for reasons of interoperability combined with provision of open source running code is necessary.

Co-opt rather than compete is the order of the day in this world.

The abuse of innovation.

2011-08-29T12:41:00.005+01:00

Innovation is a term which is widely abused and this abuse prevents us from seeing patterns in how business activities evolve.

It is difficult to see what changes when everything is called an innovation in the same manner that it's difficult to see the difference between commodification (assignment of economic value) vs commoditisation (shift from imperfect to perfect undifferentiated competition) because of the catch-all nature of the term commodification (i.e. it's used to mean both).

Take for example the utility provision of computing infrastructure (as per Amazon) - is it an innovation?

When it comes to computing infrastructure, the innovation of modern computing probably started with the Z3 in 1941. This act of innovation created an entirely new class of activity - computing infrastructure - which has evolved over time through various stages with custom built examples (LEO etc), products (IBM 650 and onwards) and eventually led to commodity and utility provision.

For reference, the full cycle is innovation, custom built, product (with rental services) and commodity (with utility services).

Two things should be noted, firstly that the pathway of evolution is common for activities (and knowledge) though it's not a time based sequence. Secondly, the innovation of the Z3 created a new class of activity rather than evolved an existing class (as with the first phone, the first radio, the first ...)

When it comes to the shift from products to utility, this simply represents an evolution of an activity and not the creation of a new form i.e. infrastructure existed before Amazon. However, it is perfectly true to say that this evolution enables (through creative destruction) and accelerates (through componentisation) the innovation of higher order systems i.e. as infrastructure has evolved we've seen an explosion of innovation in big data, mash-ups etc. This is perfectly normal as commoditisation (the common term used to describe this evolution) creates a cycle with innovation.

So, we have a difference between innovation of a new activity and evolution of an existing activity - both of which we unfortunately call innovation.

To complicate matters there's also the consumer and provider perspective. Whilst electricity is a commodity provided through utility services to consumers, behind the interface (the plug) has been a world of innovation of novel activities (wind farms, solar power, geothermal etc) aiming to create some form of operational advantage. However, it is worth noting that this provider innovation doesn't suddenly turn a consumer commodity into an innovation.

Finally we have terms like sustaining and disruptive innovation. As an activity evolves, in many cases changes to the activity (such as feature differentiation in the product stage) are sustaining and occasionally they are disruptive.

When an activity evolves across a boundary i.e. shifts from products to utility services (as with cloud) then this shift is generally disruptive because the incumbents have huge inertia to the change caused by their past success in the previous stage of evolution (i.e. product or rental vendors).

So the pattern we have is :-

Innovation of a new activity which is distinct from the evolution it enables.
Evolution of an activity from innovation, custom-built, product (rental) to commodity (utility services). This process is commonly called commoditisation.
Sustaining changes dominating within domains (i.e. product)
Disruptive changes dominating between domains (i.e. product to utility services)
Enablement and acceleration of the innovation of higher order systems through commoditisation of lower order subsystems (i.e. creative destruction and componentisation)
A difference between consumer and provider perspective.

Now, the problem with the abuse of the term innovation is we end up with :-

Innovation
Innovation
Sustaining Innovation
Disruptive Innovation
Innovation
It's my product, of course it's an innovation ...

You have no hope with spotting the pattern under such circumstances and it's no wonder that people get confused with this subject. This has severe impacts on management practices but that's a post for another day.

As for Amazon's EC2 it represents an evolution of an existing activity which is disruptive, will enable innovation of higher order systems and for the provider has probably involved some innovative pursuits.

I hate to give up on words, however "innovation" has become so widely abused as to be meaningless. For the future I'm tempted to use the word "Genesis" to describe the creation of a new activity and to put "innovation" in my book of pointless words along with "Cloud" etc.

Hosting Con Keynote

2011-08-18T16:23:00.004+01:00

I was very fortunate to be asked to give the opening keynote at Hosting Con 2011 covering commoditisation, business evolution, leadership and what the various tactical plays in the cloud computing space mean to hosting companies. The audience was fantastic, I had a great time and despite using excessive numbers of slides, no-one was hurt in the process.

Continuing on the theme from my OSCON tutorial, I've uploaded a summary set of slides which are highly condensed but give a taster to what we covered.

Alas, there's no video and as per usual I'm six years into writing my book and around 30% of the way there. The subject matter keeps on giving me more areas of interest to explore, so don't hold your breath for me to finish any time soon.

Summary of Hosting Con 2011 keynote

View more presentations from swardley

OSCON Tutorial

2011-08-02T21:29:00.004+01:00

I gave a three hour tutorial at OSCON on innovation, commoditisation, business evolution, organisation, leadership and various tactical plays in the cloud computing space. The talk was a blast, I really enjoyed it and judging by the feedback it hit some home runs with many of the audience.

However, the presentation is 1,041 slides long and so - I'm not uploading that or creating a video. Instead I've made a summary presentation which covers the main points.

Be warned, it's highly condensed.

Situation Normal Everything Must Change
View more presentations from swardley.

Is Microsoft's biggest enemy … Microsoft?

2011-07-05T19:39:00.018+01:00

Last year at OSCON, I examined mechanisms by which a company could use technology evolution to disrupt an existing player with minimal fear of retaliation. To quote myself :-

"it's the incumbents existing model which will protect you"

This year at OSCON, I'll be giving a three hour tutorial which will explore the entire subject of organisational warfare in far more detail. To give a taster of what is to come, I thought I'd expand upon some of the reasonings behind the above statement.

Anyone who has been following my public presentations over the last seven years or has been exposed to my exploration of this subject over the last decade+ will be well versed in much of this practice. For those uninitiated in this field, I'll start with some basics.

All business activities evolve through a common lifecycle and Cloud Computing is simply an example of this. Unfortunately, whilst we know how things will change*, we cannot say when. The pattern of evolution is independent of time which is why the lifecycle graphs that I use have no time axis. But then they could never have a time axis, the future is an information barrier we cannot peer past.

Fortunately, there are also barriers to the process of evolution and these give us a sense of when things will happen. In order for an activity to evolve from the domain of products to that of utility services then the following four factors are required - concept, suitability, technology and change in attitude. These factors are our "clue" that change will happen.

So we can predict how things will change*, just not when - at least not with any accuracy.

[* Correction : I originally said "what will happen", however for the sake of clarity, we cannot predict what innovations will occur as that is governed by uncertainty. We can however make reasonable predictions of how existing activities that have spread will evolve just not on what time frame]

As any business activity evolves along its lifecycle its characteristics change from more chaotic (e.g. constantly changing, highly uncertain) to more linear (e.g. defined, predictable). Using this change of characteristics we can develop organisational models which cope with evolution. An example of this is the Innovate-Leverage-Commoditise (ILC) pattern which can be found with many cloud companies.

So we can predict what will happen and though we can't predict when, we can design an organisation around evolution.

There are two tactical playbacks which can be used in such an environment. The first is around creative leadership, think Steve Jobs' Apple. The other is around disruptive leadership for which the best example would probably be Amazon.

Amazon is a company which rarely seems to create a new activity but instead specialises in commoditising activities and disrupting existing players. Infrastructure existed before EC2, book distributors existed before Amazon.com and the paperback existed well before the kindle. Amazon is extremely good at the disruption game.

To illustrate this disruption play further, figure 1 provides a rough tactical map of Salesforce. Whilst the incumbents are firmly entrenched in the product world for provision of CRM (& sales automation), Salesforce has commoditised this activity through provision of a standardised service. It also appears to be operating an ILC pattern i.e. core services around which a growing ecosystem is used to encourage innovation and identify new successful patterns which are then subsequently commoditised to core services - hence innovate, leverage and commoditise. This model is little different from Amazon's play in the infrastructure space.

Figure 1 - Tactical Map of Salesforce (click on image for higher resolution)

On closer inspection, Salesforce seems to be doing more than just commoditisation with an ILC pattern, as can be clearly seen from Radian's 6 acquisition. They also seem to be operating a tower and moat strategy, i.e. creating a tower of revenue (the service) around which is built a moat devoid of differential value with high barriers to entry. When their competitors finally wake up and realise that the future world of CRM is in this service space, they'll discover a new player dominating this space who has not only removed many of the opportunities to differentiate (e.g. social CRM, mobile CRM) but built a large ecosystem that creates high rates of new innovation. This should be a fairly fatal combination.

But, how is Salesforce able to get away with this? Why was it that Amazon and not a hosting company created this future world of infrastructure provision? The answer would appear to be … inertia.

The existing players in the CRM world are stifled by their very own success in the product world. This past success creates an inertia barrier to change. Whilst the causes of the inertia barrier can be traced back to the early stages of a companies formation, by the time a company is of a reasonable size it is often embedded in the organisation, culture and reinforced by external financial markets. Figure 2 provides an overview of this.

Figure 2 - Causes of Inertia (click on image for higher resolution)

So, back to the question about Microsoft. Whilst Microsoft is now making some strong moves into the service world, the problem for MSFT is this space is being even more aggressively commoditised through open source. Whether it's VMware's necessary play into open source platform with CloudFoundry or the Openstack effort to out-commoditise Amazon.

Both efforts exploit the natural end state for ubiquitous and well defined IT activities i.e. good enough components provided through a marketplace of providers based upon common open source reference models. Both efforts will seek to create higher order revenue streams such as assurance, exchange, marketplaces and brokers alongside the normal business of being a service provider. As per my earlier point, whilst we can't say when this will happen, we can say that the factors are already in place and so it's likely to happen soon.

Whilst MSFT has made much of a fanfare about its recent moves into the cloud, it was a probably a significant internal battle for MSFT just to make the change from products to services. However, this new world is likely to be rapidly commoditised to marketplaces based around open source and hence the real question becomes whether MSFT will be able to make the further change necessary to survive in that world?

Microsoft's future business should be intertwined with open source in the domain of utility services. Unfortunately, the last group of people who are usually willing to accept such a change are those who have built careers in the previous domain e.g. products. The bad news for Microsoft is that group probably includes a large chunk of its own organisation. Hence Microsoft itself is probably its own greatest threat to future survival.

Or as the great Bill Gates once noted:-

"Success is a lousy teacher."

That's one of those basic lessons which often gets forgotten in business. In this world of competition, there are two fronts to fight on. The external front includes those competitors who attempt to either gain a creative leadership position or to disrupt your existing model. The other front is internal and against your own past success.

Which is why in my latest research I've been looking into the web 2.0 world to see if we can't find techniques, strategies and methods for managing IT and the business more effectively in a continually changing world. Of course, I already know that there exists significant competitive advantage in organisational design and the application of cybernetic management as was demonstrated through my successes, failures and experimentation with Fotango during '01-'07. The real question for me is how widespread have equivalent practices become and who is pushing the envelope with culture, organisation, ecosystems and a complex adaptive approach?

Responsibility

2011-06-04T18:23:00.003+01:00

I'm traveling on a research trip in the U.S. and whilst my topic is one of organisation and the impact of change, I cannot help but notice the current "confusion of responsibility" that appears to be occurring in the mainsteam media.

All business activities evolve and all organisations are in a constant competitive struggle. One of the impacts of this change is that previously successful models are often replaced, for example the current shift from software products to software services. Unfortunately, organisations who have built around the previous model often face internal barriers which create an internal inertia to making the change necessary for future survival. Without this inertia, external disruption would not occur and today's giants of industry would generally be tomorrow's giants of industry.

Disruption is simply a consequence of management failure, it is the responsibility of management to ensure the organisation not only survives and competes today but also tomorrow. However, management doesn't operate in a vacuum and the inertia is often exacerbated by the actions of the financial markets i.e. it becomes difficult to change a successful model even though that model has a limited lifespan precisely because the financial markets are short term and see only short term value.

The pressure that the financial markets create is immense for any large organisation and ultimately those financial markets share a heavy burden of responsibility for failures caused by the pressure they exert. However, the short term view of the markets is such that this is not seen as their responsibility and long term failure is solely put down to management.

Unfortunately, the financial markets are very adept at blaming others for what is ultimately their fault. The current economic crisis caused by irresponsible lending, a merry go-round of debt re-capitalisation and excessive exposure are all the responsibility of the financial markets. Whilst you can point fingers at consumers, that's like saying a toddler shouldn't have eaten the candy you gave it and blaming the government is like blaming a parent for not providing some rule to govern your irresponsible choice.

The financial markets are entirely responsible for the economic crisis. They're equally responsible for many company failures through short-term attitude. They're also responsible for acting in a dishonorable manner when it comes to the Fed Bank - through the exploitation of quantitative easing for financial gain to the using of Fed money to recapitalize rather than increase lending.

The U.S. Gov and Public should realise that the Financial markets act on a short term basis without consideration of the long term consequences. They act with the persona of an adolescent, always blaming others, threatening to run away, demanding less rules and pointing to a lack of rules as a reason for their own reckless behaviour. They discount the long term to a frightening degree.

Short of grounding the child (the fiscal equivalent of nationalisation), the Gov should understand that the financial markets are an offspring of society and despite the threats they have to operate within its rules. Those rules should and can be gamed to encourage more longer term responsible behaviour.

The financial markets won't magically learn responsibility, that behaviour has to be taught and encouraged. Left alone, they'll just get upto the same old tricks.

Open source as a tactical weapon, VMware's latest move.

2011-04-12T19:23:00.006+01:00

All business activities evolve through a common lifecycle and we're currently witnessing a shift of many IT related activities from a product to a utility service world. This is commonly referred to as "the cloud". This transition brings benefits, risks, different methods of operating but also impacts the tactical plays in the great skirmish between companies. There are two models of tactical play which are particularly noteworthy - ILC and Tower & Moat.

In my previous LEF post, I discussed the innovate, leverage and commoditise (ILC) model that seems to be naturally appearing in companies such as Salesforce. To summarize, it is a technique by which a company uses a surrounding ecosystem to not only reduce the cost of innovation but to encourage innovation and rapidly identify success. By acquiring and providing such innovations as common services, a virtuous circle can be created.

A second model is the tower and moat. The principle here is to defend a revenue stream (the tower) by creating a moat devoid of differential value with high barriers to entry around it. By way of example, Salesforce created its own tower around provision of CRM as a utility service in a world where CRM was generally provided through customisable products or rental services. As barriers to entry into this new field were eroded (i.e. Amazon enabling widespread access to utility infrastructure) then new barriers were created through the acquisition of platform technology.

Whilst product based competitors attempt to differentiate themselves with activities such as social CRM, Salesforce acquired such activities with the view of providing common services. The net effect is this eliminates the differential value of social CRM and helps establish a moat. Salesforce has been extensively using its ecosystem (an ILC model) to identify and acquire a wide range of potential differentials and further strengthen its moat.

When competitors finally move to a cloud model then they will find the space inhabited by a large player with a large ecosystem and few opportunities to differentiate - a reasonably fatal combination.

Both ILC and the Tower & Moat model are powerful tools which can also be used to counter competitors. They can be used together, or individually or combined with other tactical plays such as open source.

Take the case of Apple vs Android: whilst the iPhone is not one activity but a device describing many activities, Google has effectively created an ecosystem around Android which provides a means of identifying and accelerating innovation in this field whilst reducing costs. By providing the system as open source and creating a hardware ecosystem, then Android has effectively removed much of the differential value that Apple might have sort. Apple would appear to have been pushed into a high risk, stand alone innovation game against a broad ecosystem.

Take the case of cloud infrastructure: we've already seen Rackspace & NASA move to create open source software - the OpenStack project - to provide infrastructure as a service. Their vision is to create a competitive marketplace of computer utilities around OpenStack. Such a world plays to Rackspace's strength of service delivery as a utility provider but also fits with NASA's goals of increasing efficiency of infrastructure. The ecosystem around openstack should encourage rapid innovation and if successful will create the standard that a competitive marketplace depends upon. It will also drive out differential value in this space making it tough for new competitors or those with a proprietary offering.

Take the case of large scale infrastructure: into which Facebook has announced the OpenCompute project and in effect open sourced how to create large scale data centres. This should over time help eliminate differential value that such knowledge created and whilst beneficial to the future computer utility world it will also help to undermine those for whom such skills have acted as a barrier to entry into their industry - namely massive scale search engines and data processors.

Take the case of healthcare: which has seen the VA (Veterans' Association) create an open source electronic health record system from VistA. It seems clear that the VA are focused on encouraging innovation through ecosystem effects and creating a marketplace of competitive providers. Visions of a worldwide standard are not beyond the realm of reason.

Take the case of platform as service into which VMware has announced an integrated set of open source platform components known as CloudFoundry. If successful and there's every reason to believe it will be then VMware will succeed in creating a huge moat devoid of differential value in the platform space and a vast ecosystem driving this. Any would be competitors will face an uphill struggle to compete against VMware's effort. Those planning proprietary platform offerings should take note of this move.

But wait ... where's the tower?

The beauty of creating a competitive marketplace of utility service providers is that it opens up a huge range of opportunities from service provider, support, assurance, brokerage, exchange, marketplace and a dozen more. Being at the heart of this, which is where VMware will be, means they are well positioned to take advantage. It's a bold move, perfectly timed and well executed.

Of course, CloudFoundry has already been made to run on Amazon EC2 which means CloudFoundry on OpenStack built on an environment designed around OpenCompute can't be far behind.

The world of IT is changing and many IT activities have become suitable for provision through utility services. With this change comes tactical plays designed to take advantage of this shift. At the OSCON conference in July 2007, I stated that in this future utility world open source was the only way of effectively competing. Time will tell but the increasing drive towards open source and its use by major companies as a tactical weapon seems to be pointing that way.

Smart move by VMware, it'll certainly shake up the industry.

A question of standards.

2011-04-11T12:25:00.004+01:00

I'm all in favour of standards for the "cloud" world because such standards should enable and accelerate innovation in IT through componentisation effects as well as encourage the formation of competitive markets of compute utilities

However, that said, I'm against standards committees and the concept that open standards (as in APIs & data formats) are enough to create the portability required for competitive markets.

In the former case, the market will decide the standard and the job of any standards body should be to rubber stamp what is an existing practice. Unfortunately, standards committees are often used as vehicles to promote specific vendor interests and in many cases their efforts are counter productive. For example, whilst IPX/SPX was the committee approved standard, it was TCP/IP which won the marketplace battle. The only impact that approving IPX/SPX as a standard had was to temporarily slow adoption of TCP/IP in some quarters and hence inhibit innovation.

Standards should have a positive effect but defacto has to precede dejeure. There are some exceptions to this but they are exceptions.

In the case of open standards, a competitive marketplace requires multiple providers, access to code and data (ideally with syntactic interoperability) and semantic interoperability of services. Whilst open standards provide part of the solution, it is critical for reasons of semantic interoperability that a common reference model (i.e. running code) is provided.

Since, we're talking about a world of service (and not feature) differentiation for activities which are fundamentally commodity by nature and hence suitable for utility service provision, then the obvious solution is an open source reference model as the standard. Potential examples of such would be the OpenStack effort.

Such open source reference models would ideally exploit the dual nature of GPLv3 which is both restrictive in the product world but simultaneously permissive in the service world to create a functioning marketplace. Unfortunately, many vendors promote open standards (as in APIs etc) as the solution to these problems of portability and hence describe their systems as open when they're quite clearly not.

So, in general :-

Are standards good for the cloud?
Absolutely, it'll encourage innovation through componentisation effects.

Are standard committees good for the cloud?
Generally no. At best they should rubber stamp market chosen approaches however in reality they're more likely to get in the way or slow progress by promoting vested interests.

Is portability between providers important for the Cloud?
Absolutely, it's the route to formation of competitive marketplaces and reducing outsourcing risks.

Will open standards provide the portability needed?
Only in the most trivial cases but not for the vast majority of activities. Open standards are necessary but they are not sufficient to provide the portability required. The idea that open standards alone will achieve this will inhibit the formation of competitive markets.

Is open source essential for the cloud?
Absolutely, the formation of competitive markets without loss of strategic control to a specific vendor depends upon the provision of open source reference models as the standard.

Preparing for war

2011-03-29T19:04:00.012+01:00

Following on from my post on ecosystem wars, I thought I'd discuss some techniques and tactics to use. However, before I do, I need to first describe the organisation that I hope you work for. If this doesn't describe you and your organisation then the techniques and tactics probably won't make sense and I can save you the trouble of reading the next post.

Once again, my apologies to regular followers or attendees of my presentations, this post is just scene setting and so it runs the danger of teaching grandma to suck eggs.

I'm going to assume you're battle hardened, you realise that business is all about warfare and you've got a keen eye for competition. You've probably done a stint in the typical organisation, scratched your head over some of the practices and ultimately found yourself in a new sort of company (possibly created by you).

You understand what an organisation is. Rather than making the mistakes of the past and simply grouping people, activities and methodologies into generic structures such as IT, Finance, Marketing - you've taken the time to look. You may well have profiled your organisation (see figure 1) and identified those activities you consume, those activities you sell and those activities which act as barriers to entry into your industry.

Figure 1 - Profile (click on image for higher resolution)

You know the importance of ecosystem and how this can both accelerate innovation and reduce your risks. You might be using a model akin to ILC (innovate, leverage and commoditise) to grow and keep a vibrant ecosystem around your products and services (see figure 2).

Figure 2 - ILC model (click on image for higher resolution)

You know full well that methodologies have to change with lifecycle because characteristics do and how activities are interconnected (see figure 3 & 4). One size never fits all isn't effective hence you don't engage in the typical debates of this vs that, agile vs six sigma, push vs pull - you know you need both.

Figure 3 - How Characteristics Change (click on image for higher resolution)

Figure 4 - How Methodologies Change (click on image for higher resolution)

You also recognise that not everyone is the same, you have different types of people :-

Your pioneers are feverishly imaginative, often chaotic in nature, constantly exploring and they create the crazy stuff. They work on a cadence of weeks, maybe months and they fail often and miserably but they're never afraid to experiment. They want to push things out there, they're infectious, they can even be hazardous. They're not a "safe" pair of hands but you don't want them to be. Every now and then they create your future source of competitive advantage, though you don't know it until it starts to be adopted by the wider ecosystem. They are your Da Vinci's.

Your town planners build the core of your company and the components which your pioneers develop upon. They build the solid, useful and beautiful and they're obsessed with getting things better, faster, more efficient and more reliable. They're meticulous, methodological and geniuses who hide worlds of complexity behind standard interfaces. They're ultra reliable, your "safe" hands and they work on a cadence of months, even years. They provide the operational efficiency which you can use to bury your competitors. They are your rock, they are your Vitruvius'

In between, you've got your settlers. They watch the landscape, your competitors and spot the patterns which are going to either disrupt an existing revenue stream or reduce some barrier to entry. They constantly take innovations away from the pioneers and force them to move onto the next thing. They'll drive that innovation into the wider ecosystem, spin it out or fail it fast where necessary. They adapt to the changing environment and adopt what's growing. They've always got their eye on pushing a growing trend towards the town planners. They listen to the ecosystem and whilst they don't create the future or build the cities, they play the games of tactical warfare which are so essential to survival. They are ruthless to your competitors, nurturing to your own ecosystem and you're just glad they work with you. They are your Machiavelli's.

You probably identify yourself with one group and you could probably have read down the list of your employees naming which group they belong to. You didn't have to, they told you.

You've long given up on trying to create departments with a mix of Da Vinci's, Machiavelli's and Vitruvius' and listening to the constant arguments within those departments about how to best run IT or whatever they're assigned to do. You understand they won't agree, they never will.

You understand that outsourcing a department may get rid of the arguments but you'll end up removing sources of competitive advantage. It's better to let the town planners outsource those activities that can be removed.

You've given up making speeches about the need for innovation or efficiency, you know you need both. You've also realised that you need to nurture both pioneers and town planners to do this and you have to let them use the tools they need to get the job done. You understand those tools are different. One size fits all seems a long distant and best forgotten memory.

Hence you've probably decided to structure yourself around change and have groups like pioneers, settlers and town planners - though you'll call them something else. You've probably discovered the alignment issues between your old groups was an artificial construct of how you organised yourself. You've probably also realised the importance of the settlers in managing change and wondered why you didn't do this before.

You've almost certainly realised you want the best Da Vinci's, Machiavelli's and Vitruvius' to compete and hence focus almost religiously on high levels of talent and acquiring the best, no matter where they are in the world. You remove any and all unnecessary vestiges of the old world - expenses, timesheets, standard company laptops and so forth and instead have instilled a culture of respect.

You're going into a war. You want your people fighting and you want them to have the tools they need. if you can't trust your people to fight on your side then you don't want them. Your management reflects this, your organisation reflects this, your people reflect this. It feels vastly different from the old ways you remember.

If this is you, then the next post on tactics will make sense and we can go on to explore some of the actions being taking by the big players in our ecosystems.

Ecosystem wars

2011-03-29T02:26:00.011+01:00

Back in 2005, I started to talk publicly about lifecycle concepts and the importance of ecosystems. In the next two posts, I'd like to draw a line in the sand and cover those basics of ecosystem one last final time.

I'll take the liberty of assuming notions such as lifecycle, organisational profile, componentisation and consumerization are universally understood and start with an explication of what a business is.

What is a Business?

A business is a living thing, comprising a network of people, a mass of different activities, and reserves of capital including financial, physical, human and social. It consumes, it produces, it grows and it dies. Like all organisms, any business exists within a number of ecosystems in which it competes and co-operates with others; it’s shaped by and shapes its environment, and hence needs to adapt constantly merely to survive.

People come and go, activities change, and hence all firms are in a constant state of flux. In any industrial ecosystem, new activities (innovations) are a consequence of competition and those that are useful will diffuse throughout the ecosystem becoming more of a commodity. This constant change creates a paradox, identified by Salaman and Storey:

”Survival requires efficient exploration of current competencies and ‘coherence, coordination and stability’; whereas innovation requires discovery and development of new competencies and this requires the loosening and replacement of these erstwhile virtues.”

These two extremes of survival (today and tomorrow) have diametrically opposite concerns, and the techniques, tactics and methods needed to manage each are entirely different. Those who manage organizations are therefore caught on the twin horns of a dilemma: how is it possible to be standardized and efficient as well as innovative and new, without prejudicing your survival - either today or tomorrow?

The effects of this on business can be seen in the constant restructuring to cope with new paradigms, and in the yo-yoing of popular management theories between opposites in a scramble to maintain order. A more effective balance can be found through embracing both goals simultaneously.

How do we balance both goals?

This requires a rethinking of how we organize, and a realization that what really matters is not innovation or efficiency per se, but how we continuously manage the path between the two. To explain why this is the case, let us examine a typical profile of an organisation (see figure 1) and consider how we build systems for the future.

Figure 1 - Using profile to build systems (click on image for higher resolution)

First, let us take those cost of doing business activities which act as underlying components of other business activities such as payroll, computer infrastructure, authentication etc. All of these activities are linear, well defined, ubiquitous and suitable for provision as standard re-usable components through utility services. Such an approach will increase our agility, rate of innovation (componentisation effects) and reduce the cost of gambling for any innovative activities built upon these utility services.

However, it is important to understand that all innovations (i.e. those activities which are uncertain and rare) are a gamble and whilst we can reduce costs we can never eliminate it. The future value of something is inversely proportional to the certainty we have over it, we cannot avoid this information barrier any more than we can reliably predict the future. However, there is a means to maximise our advantage.

By making these utility services accessible through APIs, we not only benefit ourselves but we can open up these components to a wider ecosystem. If we can encourage innovation in that wider ecosystem then we do not incur the cost of gambling & failure for those new activities. Unfortunately, we do not enjoy the rewards of their success either.

Fortunately, the ecosystem provides an early warning mechanism of success i.e. adoption. Hence by creating a large enough ecosystem, we can not only encourage a rapid rate of innovation but also leverage that ecosystem to identify success and then either copy (a weak ecosystem approach) or acquire (a strong ecosystem approach) that activity. This is how we maximise our advantage.

To capitalise on this, we simply drive our newly acquired activity towards utility service provision and create the next wave of innovation through further componentisation effects. In this manner we create a virtuous circle of encouraging innovation in the ecosystem, leveraging the ecosystem to identify the next pattern and commoditising the pattern to utility services in order to encourage the next wave. I've summarised these concepts in figure 2

Figure 2 - ILC model (click on image for higher resolution)

What is critically important to note, is that in essence both innovation of new activities and efficiency of commodity provision can be outsourced. The innovation of activities can be outsourced to a surrounding ecosystem building upon our services, whilst those commodity activities can be outsourced to a marketplace of utility providers. There are benefits to retaining some element of control in those areas but the critical area to focus on is the transitional phase and how you leverage the ecosystem. Our ability to exploit the link between innovation and commodity depends upon the size, composition, engagement and overall level of activity within that ecosystem.

Whilst each company comes with its own personal ecosystem (its staff, its partners) growing an extended ecosystem and using that to manage change is a powerful tool and a competitive weapon. It's not business but ecosystems that collide in the commercial world and woe betide that organisation which has the weaker. We've seen this story repeated many times before at many different levels through all the usual examples of BetaMax vs VHS or TCP/IP vs IPX/SPX.

It's imperative to understand that a well formed and large ecosystem leads to lower costs of innovation, creates higher rates of innovation, improves the rates of successful adoption and encourages greater efficiencies. If it is just you and your employees then you'll find it difficult to survive a direct onslaught from any of the modern day monsters with well developed and extended ecosystems (such as Google, Amazon etc). Not impossible, just difficult.

For more details, my OSCON '10 talk provides a useful overview of these concepts. However this is enough of a base to begin with.

In the next few posts, I'll cover some of the consequences of this and the strategies which are commonly deployed. Again, I realise this is nothing new and apologies to all readers who've had to endure me harping on about this over the last five years but I'm just scene setting for much later posts.

Consumerization

2011-03-28T17:19:00.006+01:00

The evolution of any business activity is complicated by the existence of multiple ecosystems such as the enterprise and consumers. Activities can exist in, transfer between and commoditise within any of these ecosystem.

For example, whilst CRM has mainly commoditised in the enterprise space, email has crossed over to the consumer space in which it has been more aggressively commoditised. Today, these heavily commoditised mail services (such as Google Mail) create adoption pressure for the enterprise. These concepts are summarised in the figure below.

Figure 1 - Consumerization & Lifecycle Across Ecosystems (click on image for higher resolution)

When discussing an activity, a key issue to consider is in which ecosystem the evolution of that activity is governed i.e. is it being driven by competition in the enterprise or the consumer space. Back in 2001, Douglas Neal and John Taylor described the process by which governance transfers from the enterprise to the consumer ecosystem as Consumerization.

The consumerization of technology and its subsequent commoditisation in that consumer ecosystem can have a profound effect on the operation of any enterprise using that technology. It should be noted:

Not everything transfers. Some activities find meaningful use within only one ecosystem and hence tend to develop only in that environment. For example, CRM.

Barriers to entry can erode quickly. The impact on the media industry of the commoditisation of the means of mass communication occurred primarily in the consumer space over a single decade. Originally, the principal means of mass communication was provided through printing presses and supply & distribution chains. These acted as a major barrier to entry into those industries. The internet and digitization of content brought new means of mass communication into the consumer ecosystem and subsequently were aggressively commoditised resulting in widespread use of blogs, wiki and social media tools such as twitter. A consequence of this is that the barriers to entry into the media industry have been completely eroded. Anyone can become a digital newspaper.

Ecosystems are not isolated. People are part of multiple ecosystems, hence activities in the enterprise (which may be provided through products) tend to start with custom-built solutions and hobbyist equivalents in the consumer space.

The rate of commoditisation is not uniform between ecosystems.

Numerous activities have undergone this change. The pressure for adoption created through consumerisation has created positive responses such as BYOT (bring your own technology) and negative responses such as shadow IT i.e. when an organisation fails to deliver what its internal consumer need (Joe Baguley gave an excellent presentation on this topic).

All organizations should, as a matter of course, examine both their own and the consumer space for the growth of highly commoditised activities and consider the impacts of these and whether barriers to entry into their own industry will reduce.

For point of reference, computer utilities (i.e.Cloud computing) is expected to reduce the barriers of entry into many industries. In such cases an approach of adoption of the service and creation of higher-order systems can be used to re-establish those barriers.

Equally the impact of consumerization on the means of manufacture is going to hit a wide range of industries. This video is a foretaste of what is going to happen.

Building a private cloud

2011-03-18T10:45:00.004Z

As we move from a world of infrastructure products to a competitive market of infrastructure services, private clouds have a valid role as part of a hybrid strategy in mitigating transitional risks (such as concerns over data governance, trust, transparency and security of supply).

The key word here is transitional i.e. this is not a permanent solution. Furthermore by using a hybrid strategy you're deliberately sacrificing some element of the benefits of public provision (economies of scale, focus etc) in order to mitigate those risks. It's a trade-off.

However, for this post I'll assume you've decided to make that trade-off. Hence you're going to be using a combination of public and private resources in a hybrid form for the immediate future. So dealing specifically with infrastructure, at what price should you be building a private cloud?

First, you need to recognise that a consequence of a shift to standardised components provided through utility services will be an explosion of innovation in higher order systems - i.e. the competitive landscape is going to become even more competitive. Second, cost efficiency won't result in reduced IT budgets because increased competition, the long tail of unmet demand and co-evolution effects will result in far greater consumption. Lastly, one of the key benefits of cloud is speed and when this is combined with cost efficiency, you'll end up doing more, faster with the same budget.

It's worth noting that there is little correlation between total IT spending and business value but there is strong correlation between speed and business value. How efficient your provision is in terms of speed is critical here. Now assuming you're not planning something daft like a long winded requisition process for new virtual machines, then the speed of provisioning will be minutes whether public or private. Assuming this is uniform, the distinguishing feature becomes cost efficiency - i.e. how much MORE you can do with same level of spending over competitors.

With public provision such as Amazon EC2, a small instance (I'll refer to this as a VM) costs currently $740 per year. Now whilst Amazon doesn't release figures, we believe the cost of provisioning such a VM can be much lower, possibly as low as $220 per VM per year. Hence if your private cloud is being designed to cost $1500 per VM per year, you might be forgiven for thinking that this is only twice Amazon pricing and represent good value for a hybrid trade-off. What you're probably discounting is that your private environment will have to exist for three or more years and during that time Amazon's pricing could fall rapidly, possibly to the order of $300 per VM per year.

In such circumstances, a competitor with a pure public play would have upto a 5x cost advantage (depending upon how much of your hybrid strategy was private provision) which is an astronomical difference in a world of increasing and more rapid competition. But that's a key point to note, the impact depends upon the type of industry you exist within and your competitor actions.

Assuming you're working in a high tech reliant industry, then in order to give your private cloud a fighting chance, you have to be designing for a target price of between $220-$540 per VM per year.

These figures (and lower) are achievable if done at reasonable scale and with a relentless focus on commodity provision. For reference purposes only, I've provided a hypothetical example (based upon real data from real examples) of a VM farm. I've used the following assumptions:

Base VM: 1.7 Gb RAM, 160 Gb HD, CPU equivalent to a passmark of 420
Target Price (VM per year): $510
Max Utilisation (to determine level of over-provisioning required): 70%
No. of VMs: 100,000
Base Unit: Rack
Focus: Commodity (No multiple PSU, Hot Swap)
Depreciation for hardware: 3 yrs
Current High Interest Rates of: 3%
Networks: All internal equipment and network upto external routers but not external bandwidth.
PUE:1.3

Given this, then figure 1 shows as a rough guide, how your target price per VM per year breaks down into its various components.

Figure 1 - Cost per VM per year.
(click on image for larger size)

Now obviously there are many ways to skin a cat, and pricing is sensitive to various factors such as scale, density of VMs, power consumption, number of units per rack, location etc etc. However, I put this here to start a discussion as I would like to know what's your cost per VM in your private cloud?

Componentisation

2011-03-07T10:23:00.008Z

Herbert Simons showed in the Theory of Hierarchy how the evolution of a system is dependent upon the organisation of its subsystems. In short, as an activity becomes commoditised and provided as ever more standardised components, it not only allows for increasing speed of implementation but also rapid change, diversity and agility with systems that consume the activity as a subsystem.

In other words, the creation of standard sizes of bricks, pipes, utility services and other architectural building blocks led to a faster rate of house building and a wider diversity of housing shapes. It's the same with electronics and every other field you care to look at.

Commoditisation to standard components leads to an explosion of innovation for higher order systems.

The cloud is no different. It's the creation of good enough, standard components which will cause a wide diversity of higher order systems. These will in turn undergo their own evolution to more of a commodity. Without this process we would never have got to a CPU, let alone cloud or the hugely complex systems that will develop from it.

This also doesn't mean that innovation stops with the standard components. Whether it's brick making or electricity provision, there is a huge amount of operational innovation hidden behind the "standard" however the "standard" acts as an abstraction layer to this. Just because my electricity supplier has introduced new sources of power generation (wind turbine, geothermal etc) doesn't mean I wake up one morning to find that we're moving from 240V 50Hz to something else. If that constant operational innovation was not abstracted behind the standard then all the consumer electronics built upon this would need to continuously change - the entire system would collapse in a mess.

Whilst cloud computing is all about commodity provision of computing resources through utility services, these components act as the abstraction layer. We're not quite there yet, we're still in transition from a product to a utility service world. However, even in that utility service world, providers will still innovate in terms of operations but to a consumer that will be abstracted and ultimately reflected in better price and quality of service. To the consumer this will be seen as operational efficiency and the activity itself for all sense of purpose will remain a commodity, just like electricity and every other industry that has undergone this change. Of course, I'll do more things with it, create new and ever more complex higher order systems but my innovation is based upon the activity being provided as a commodity.

I can understand how many people don't relish the prospect that some activities within IT are becoming more of a commodity. They may not realise the benefits this will create in terms of progress but at the same time as they cry this isn't happening they happily consume systems which depend upon this process.

If you really don't like the idea of commoditisation and componentisation then try getting by without using anything which has depended upon it. Try building a computer without a CPU, electronics, electrical power supply or even nuts and bolts.

The online grocery guide to keeping me happy

2011-03-06T20:32:00.009Z

There are many things which I find annoying about shopping online, the most irksome generally deal with grocery shopping. Now let's assume you're an online grocer and you've covered all the basics of having stock, delivering what you say you're going to deliver, the produce being in a reasonable condition and it arriving on time (a tall order for some).

Then, my advice to you is to apply what I call the Wardley Happiness Equation of Extreme Exceptionalism (wheee! for short). The equation is trivial to apply and will almost certainly guarantee my happiness with the shopping experience.

Wardley Happiness Equation of Extreme Exceptionalism

If I'm spending over £70 with you, then you can assume that this is part of a weekly shop. Hence when you come to tot up the final prices, simply calculate the delivery time plus 7 days and then for each item discount 15% for every day which the best before date is less than this.

Please don't ask me to check the dates or tell me that lettuce has a shelf life of three days - it's nonsense. There is nothing that annoys me more about online shopping than a week's worthy of grocery turning up with some best before dates of tomorrow.

The confusion of choice

2011-03-06T10:53:00.013Z

I've been asked many times whether there are any mechanisms to slow down, prevent or reverse commoditisation? Well, there are a long list of techniques which can be employed and to explain them let's just revisit the lifecycle approach.

Figure 1 provides a typical lifecycle for an activity from innovation to commodity. It should be noted that what drives an activity through its lifecycle is both user competition (the drive for something which is useful i.e. has utility) and supply competition (the drive to provide a better way of meeting users need over competitors).

Figure 1 - Lifecycle
(click on image for larger size)

Furthermore as an activity transitions from one phase to another such as custom built to products or products to utility services, a number of factors are required for the transition to occur. The factors include:-

Concept : e.g. the idea of providing something in that format such as computing infrastructure through a utility.
Attitude: a willingness for consumers to accept the activity in the new format.
Technology : the technology to actually achieve this.
Suitability : the activity to be widespread and well defined enough to be suitable for provision in that format e.g. a utility format requires a well defined and ubiquitous activity capable of supporting the volume operations needed.

So how do we counter this?

Don't Sell it to Everyone: One of the simplest ways of ensuring that something doesn't become a commodity is to not sell it to everyone i.e. prevent it becoming ubiquitous and well defined. Unfortunately competition applies here, so you have to find a way of preventing others from selling - i.e. patents give a time limited opportunity to do this.

Create Confusion: If you can't prevent everyone selling, then create a confusion of choice. The purpose of this is to reduce certainty over the activity in order to persuade consumers that it is more of an innovation. The most profitable scenario is to have every consumer buying it and believing it is highly innovative to them alone - the "designed for you" meme.

Examples of confusion of choice can be found from light-bulbs to mobile phone contracts. Walk into your average electrical department store, go to the light bulb counter and ask yourself - "Do I really need a choice between 200 lightbulbs and bewildering array of styles, plug formats, power outputs, swirly bits of glass, shapes etc? Didn't we use to have less choice and wasn't it easier to make a decision then?"

This abundance of choice is not designed for your convenience but to prevent you from making comparison. It's the same with the plethora of designed for you mobile phone contracts or insurance instruments or ... the list is long. Why else do you think we now have a vibrant price comparison industry countering this.

Association: one of my favourite examples of manipulation is to try and install a hidden meaning to an activity i.e. to create an association between this particular brand and either a lifestyle choice or some other implied benefit. If you're lucky, turning an ordinary activity into a status symbol can reap huge dividends. This sort of manipulation can be most easily recognised by switching on the TV and watching adverts - most of which rarely discuss the activity itself but imply a consequences of choosing this particular brand. Using this brand of soap will transform you into a glamorous person, whisked away to shower in a tropical forest surrounded by other glamorous people? The use of this deodorant will make you instantly appealing to huge numbers of the opposite sex? The purpose of these acts is not to cause confusion but to fundamentally add a hidden utility to a particular brand of the activity and to disassociate it from the rest.

Complexity: Another tactic is to reduce certainty through the introduction of visible engineering i.e. make the device look more complex than it is. It's actually a combination of confusion of choice, association and complexity which turned the humble vacuum cleaner into a space age construction. It doesn't matter that the activity is ubiquitous, well defined and commodity like nor that the latest range of cleaners may be no better at cleaning. Today, vacuums have new properties such as "doesn't lose suction", they use visible engineering and a bewildering array of choice and associations to lifestyle. All of this is designed to persuade consumers that there's more to a vacuum cleaner than cleaning the floor. Kerching!

Bundling: Another favourite of mine is to bundle many activities together. For example, the iPhone is not a phone but a wide range of activities bundled into a single device - calculator, phone, camera, calendar, alarm clock, browser, email, time recorder, map etc. Bundling provides ample opportunities for confusion and dissociation, i.e. you can't easily compare one smart phone to another and certainly not the pricing tariff of a standard phone to a smart phone. However, through Android, Google has aggressively commoditised most of these bundled activities with the use of open source and open ecosystems and hence cornered Apple into a very high risk, constant innovation game. So as well as tactics to counter commoditisation, you can also use counter tactics to force it and your competitors into tricky positions.

I've always been a big fan of Apple, however this is the second time in my opinion that they've been seriously outmaneuvered and the root causes of this appear to be the same - a poor understanding of change and how to play the game. They're great at innovation but that's only part of the battle. Time will of course tell, but I don't see Apple's future as rosy.

I won't go on as there's a whole mass of techniques and counter techniques in this space. However to the question can you slow or reverse commoditisation - well you can certainly persuade consumers of this.

Private vs Enterprise Clouds

2011-02-28T08:52:00.024Z

There is a debate raging at the moment between different types of clouds hence I thought I'd stick my oar in. First, let's clear up some simple concepts:-

What is cloud?

Cloud computing is simply a term used to describe the evolution of a multitude of activities in IT from a product to a utility service world. These activities occur across the computing stack.

These activities are evolving because they've become widespread and well defined enough to be suitable for utility provision, the technology to achieve this utility provision exists, the concept of utility provision is widespread and there has been a sea-change in business attitude i.e. a willingness to accept these activities as being a cost of doing business and commodity-like.

There is nothing special about this evolution, it's bog standard and many activities in many industries have undergone this change. During this change, the activities are in a transitional phase from one model to another which creates a specific set of risks around trust in the new providers, transparency, governance issues and concerns over security of supply. There are other risks (including outsourcing and disruption risks) but these are irrelevant for the specific question of private vs enterprise clouds. For clarity's sake, I've summarised this evolution in figure 1.

Figure 1 - Lifecycle and Risk.
(click on image for larger size)

Why private clouds?

Private clouds (where the service is dedicated i.e. "private" to a specific consumer) are generally used as part of a hybrid strategy which combines multiple public sources with a private source. The purpose of a hybrid strategy is simply to mitigate transitional risks (concerns over governance such as data governance, trust etc). This is a normal supply chain management tactic and occurs frequently with this type of evolution. Even within the electricity industry you can find a plethora of hybrid examples in the early days of formation of national grids.

Fundamentally, a hybrid strategy mitigates risk but incurs the costs of both lesser economies of scale and additional resource focus when compared to a pure public play. It's a simple trade-off between benefits and risks.

Why Enterprise Clouds?

Enterprise clouds need a bit more explaining and to understand why they exist we first have to start with architecture. To keep things really simple, I'm going to focus on infrastructure.

As per above, the use of computing infrastructure has undergone a typical evolutionary lifecycle from innovation to commodity hardware and more recently the appearance of utility services. In the earlier stages of evolution, the solution to architectural problems was bound to the machine. Scaling was solved through buying "a bigger machine" i.e. Scale-Up whereas resilience involved hot-swap components, multiple PSUs and other redundant physical elements (the N+1 model). This was essential because of the long lead time for replacement of any physical machine i.e. there existed a high MTTR (mean time to recovery) for a physical server. Applications therefore developed on the assumption of ever bigger and ever more resilient machines. These architectures spread (specific sets of knowledge behave just like activities) and became best practice.

As computing infrastructure became commodity-like, novel system architectures such as Scale-Out (aka as horizontal scaling) developed. These solved scaling by distributing the application over many more smaller and standardized machines. The new architectural solution was therefore "buy more machines" and not "buy a bigger machine". This scale-out architecture rapidly spread as infrastructure became more generally accepted as a commodity.

As we entered the utility phase, infrastructure has in effect become code – that is, we can create and remove virtual infrastructure through API calls. The MTTR of a virtual machine provided through a utility service is inherently lower than its physical counterpart and novel architectures called "design for failure" have emerged that exploit this. The technique involves monitoring a distributed system and then simply adding new virtual machines when needed.

In the cloud world application scaling and resilience are solved with software whereas in legacy it was often solved by physical means. This is a huge difference and I've summarized these concepts in figure 2.

Figure 2 - Evolution of Architectures
(click on image for larger size)

By necessity, public cloud infrastructure is based upon volume operations – it is a utility business after all. Virtual compute resources come in a range of standard sizes for that provider and are based upon low cost commodity hardware. These virtual resources are typically less resilient than their top of the line physical enterprise class counterpart but naturally they are vastly cheaper.

It should be noted that a ‘design for failure’ approach can take advantage of these low cost components to create a far higher level of resilience at any given price point through software.

By way of example, the general rule of thumb is that each 9 roughly doubles the physical cost. Hence let's take a scenario of a base machine designed to give a 99% up-time with a machine designed to provide 99.9% uptime costing twice the amount etc.

In the above scenario, using four base machines in a distributed architecture provides a theoretical up-time of 99.999999% though naturally it will suffer periods of degraded performance due to single, dual or triple machine failure. In a utility computing environment, this impact is negligible due to the low MTTR of creating new virtual machines and in such as case you have a low cost environment (4x base unit), high level of resilience for total failure (1 in 100 million) and fast recovery for periods of degraded performance due to the low MTTR.

Now compare this to a single physical machine (ignoring all scaling, network and persistence issues etc). The equivalent physical machine would be 16x more costly (assuming the 2x rule holds) and in the rare situation that complete system failure occurred, the MTTR would be high. In reality, MTTR would be high for all its components unless spares were kept.

Now for reasons of brevity, I'm grossly simplifying the problem and taking a range of liberties with concepts but the principle of using vast numbers of cheap components and providing resilience in software is generally sound. We've seen many examples of this including RAID. These cloud architectures are no different except they extend the concept to the virtual machine itself.

The critical point to understand is that these two extremes of architecture have fundamentally different economic models. The cloud model is based upon volume operation provision of commodity good enough components with application architectures exploiting this through scale-out and design for failure. Now in the following figure I've tried to highlight this difference by providing a greyscale comparison between a traditional data centre and a public cloud on a number of criteria.

Figure 3 - Data Centre vs Cloud.
(click on image for larger size)

OK, now we have the basics let's look at the concept of Enterprise Cloud. The principle origin of this idea is that whilst many Enterprises find utility pricing models desirable, there is a problem when shifting legacy environments to the cloud. Now when companies talk of shifting an ERP system to the cloud they often mean replacing own infrastructure with utility provided infrastructure. The problem is that those legacy environments often use architectures based upon principles of scale-up and N+1 and infrastructure provided on a utility basis doesn't confirm to these high levels of individual machine resilience.

At which point the company has two options; either re-architect to take advantage of the volume operations through scale-out and design for failure concepts or demand for higher level resilient virtual infrastructure.

This is where Enterprise Cloud comes in. It's like cloud but the virtual infrastructure has higher levels of resilience but at a high cost when compared to using cheap components in a distributed fashion.

So why Enterprise cloud? The core reason behind Enterprise Cloud is often to avoid any transitional costs of redesigning architecture i.e. it's principally about reducing disruption risks (including previous investment and political capital) by making the switch to a utility provider relatively painless. However, this ease comes at a hefty operational penalty. The real gotcha' is those transitional costs for redesign increase over time as the system itself becomes more interconnected and used.

In principle, Enterprise Clouds are used to minimise disruptional risks but they will be ultimately subject to a transition to the new architecture because of high operational costs. There are other reasons for an "Enterprise Class" cloud but most of these, such as where data resides, can also be provided by using a "Private" cloud that is built using commodity components.

The different economic models is what separates out private / public compute utilities from enterprise clouds, I've highlighted this on the greyscale in figure 4. Actually, I prefer to use the original term virtual data centre rather than enterprise cloud because that's ultimately what we're talking about.

Figure 4 - Enterprise vs Private Cloud.
(click on image for larger size)

Do Enterprise Clouds have a future? Yes, but they'll ultimately & quickly tend towards niche (specific classes of SLAs, network speeds, security requirements etc). Their role is principally in mitigating disruption risks but the transitional costs they seek to avoid can only be done at an increasing operational penalty. It should be noted that there is a specific tactic where an enterprise cloud can have a particularly beneficial role: the "sweating" of an existing legacy system prior to switching to a SaaS provider.

Do Private Clouds have a future? Yes, they have a medium to long term benefit in mitigating transitional risks (such as issues over data governance) through the use of a commodity based model. However, be careful what you're building and remember the impact of private clouds will diminish as competitive markets develop in the public space with easy switching between providers.

Do Public Clouds have a future? Absolutely, this is the long term economic model and will become dominant through the same mechanisms that has made public utility provision dominant in other industries especially as clearing houses, exchanges and brokerages form.

Last thing, I said I would focus on infrastructure to keep things simple. The rules change when we move up the computing stack ... but that's another post, another day.

Will Cloud Computing help the business align to the market?

2011-02-26T14:17:00.002Z

I was recently at a private conference of CIOs when in the midst of discussing activities such as CRM to ERP, it became clear that not only did everyone understand these applications, but also that every company had them. Almost all of those companies have expensive customization programmes in place, tailoring these systems to fit their needs. This behaviour is puzzling, but to explain why, we'll need to first look at the concept of business evolution.

All business activities share a common evolutionary pathway, a lifecycle; from the innovation of an activity, to custom built systems implementing that activity, to the provision of products, to eventually commodity provision and the appearance of utility services.

It is unusual to find activities that are commonplace, well understood, well defined and accepted as a cost of doing business being treated as though they were in an earlier stage of evolution with heavy customization. But each CIO told how that was exactly what was happening at their company, and the discussion became surreal when we discovered that many of the customizations being made to systems like CRM were common as well.

Surely, these activities are best served by being provided as a standardized commodity ideally through a market of utility services? Isn't that the whole point of cloud computing? So why aren't we all flocking to consume a host of common activities through the utility services of cloud providers? Well, a diverse range of companies already are.

However, several of the CIOs who had looked into the issue said these services didn't work for their company. On further questioning, it became clear that it wasn't IT but the business that was pushing for the customization in order to fit in with their way of working. Several CIOs had even used the standardized services of cloud providers to challenge this - asking the business to justify the additional orders of magnitude costs for a customized system by demonstrating the differential value that 'their way' made.

Whilst we often talk about business and IT alignment, this shouldn't mean IT just delivering what the business asks for. In circumstances where there is no differential value, then it's better to 'fit to the model' rather than 'fit the model to us'. We do the former all the time - from banking to electricity provision; we don't go and build our own customized solutions, we just find a way to work with the dominant market standards.

Will cloud computing help the business to treat common, cost-of-doing-business activities as though they are common and a cost of doing business? Or, will some businesses continue to spend vast sums customizing that which really makes no difference?

Could the cloud help the business re-align itself with the market?

Reprinted from LEF site

VMware as an acquisition target?

2011-02-25T21:52:00.010Z

Back in 2009, I proposed that VMware would eventually divest itself of its virtualisation business. The reason for my thinking was as follows :-

Whilst the majority of VMware's revenue was based upon its virtualisation technology, this was an area that was ripe for disruption through two points of attack - open source systems such as KVM and the formation of marketplaces offering utility based virtual infrastructure. The latter almost certainly requires open source reference models to avoid issues around loss of strategic control and when combined with aggressive service competition around price, this doesn't leave much room for a license based proprietary technology.

A dominant position in the enterprise is no guarantee for future success - see Novell Netware and the IPX/SPX vs TCP/IP battle. Critical in such battles is the development of wide public ecosystems and inherently open source has a natural advantage. However, given the revenue position of VMware, it could not afford to undertake this route.

The obvious route for VMware would be to develop a platform play, most likely an open source route with an extensive range of value add services - from assurance to management. The current business would be used to fund the development of this approach until such time as the company could split into two parts - virtualisation & platform.

Given the likely growth of private clouds as a transitional model in the development of the cloud industry, VMware would position the virtualisation business in this space for a high value sale, benefiting from its strength in the enterprise. This is despite it being unlikely that VMware would become the defacto public standard, that the technology was likely to be disrupted, that hybrid clouds are a transitional strategy superseded by formation of competitive markets and that many "private clouds" would be little more than rebranded virtual data centres.

During this time, there would be signals of confusion over the VMware strategy precisely because it would be using a time limited cash cow to fund a new venture whilst preparing to jettison the cash cow prior to disruption.

Given the confusion over cloud, the often central (but ultimately misconceived) role that virtualisation is given in the industry, the generally disruptive effects of this change and the wealth of many competitors then in my opinion with luck, timing and good judgement a buyer could be found.

So, in my opinion:

For VMware, it would mean creating a strong platform business funded by its current revenue stream before jettisoning the virtualisation business at high value prior to its disruption.

For the buyer, it would mean ... whoops ... well that's capitalism for you. Next time, pay more attention.

Of course, this is just my opinion but I haven't changed my view over the years. I'm expecting to see an increasingly clear division within VMware between platform and virtualisation in the next year or so.

I'm hence curious to know what others think? Do you believe that VMware would sell its virtualisation business?

Should I listen to my users?

2011-02-25T16:46:00.010Z

Following on from my previous post on lifecycle, I thought I'd discuss the question of when to listen to users. I'm going to use my lifecycle and profile graphs to illustrate the points I'll be raising, so if this is not familiar to you then please read my most recent post on lifecycle.

Figure 1 - Lifecycle (click on image for higher resolution)

Figure 2 - Profile (click on image for higher resolution)

Before I start, a word of apology. All of this stuff is blindingly obvious and has been for many many years - hence please don't take offense.

Any new activity starts in the chaotic phase and then as it evolves through its lifecycle it enters a more linear phase. As it moves, its characteristics change from uncertain, deviating and a source of worth to known, standard and a cost of doing business.

Less confusion creeps in, let's just reiterate that a new activity is an innovation. Whilst we tend to abuse the term innovation, a feature differentiation is a feature differentiation, a process improvement is a process improvement and an operational efficiency is an operational efficiency e.g. you can call every process improvement, operational efficiency and feature differentiation an innovation if you want but good luck in trying to make sense of things if you do.

As explained in earlier posts, as an activity's characteristics change then the methods by which you can effectively manage it change as well i.e. we shift from agile to six sigma for example. This is why there is no one size fits all.

Equally, in the chaotic stage the approaches taken are about gambling, experimentation, potential future worth and novel practice (i.e. it's highly uncertain), however when that same activity has entered the linear stage it's all about conformity to standards, defined processes, price vs quality of service and best practices (i.e it's predictable). In between these two extremes is where ROI (return on investment) matters because unlike the chaotic stage a market exists to provide some basis for trending and unlike the linear stage, you have a choice over whether to implement it since it is not a cost of doing business.

When it comes to users then :-

In the chaotic stage the only reason why you would listen to potential users is the same reason why you might collaborate with others - serendipity i.e. the chance encounter of a better idea. Of course, whether the idea is better or not won't actually be known until you "experiment" i.e. you put it out into the market. You have as much chance as identifying the future successful innovation as any user and there are no methods of guaranteeing any innovation will be successful. Like it or not, you have to gamble. The rule of thumb is listen to yourself.

In the transition phase listening to users is essential because a market has established, users are becoming familiar with the activity, customer feedback and trending is possible and competitors can be analyzed. The rule of thumb is listen to your ecosystem.

In the linear stage, the activity is a commodity and its all about price vs QoS. Now, assuming you're not going to embark on a disinformation campaign and attempt to persuade users that a commodity is actually some form of innovation (a common tactic) then the only thing you need to really focus on is your position against competitors i.e. faster, more reliable, cheaper etc. Hence the rule of thumb is pricing & quality comparison.

So, should you listen to users? The answer is "yes and no", which one depends upon the stage of lifecycle of the activity in question.

[Added Note 26/02/11] I've just come across this Fred Wilson quote : "Early in a startup, product decisions should be hunch driven. Later on, product decisions should be data driven". It's pretty much spot on.

Deconstructing Gartner's Hype Cycle

2011-02-18T11:03:00.033Z

This is a piece of work that I did many years ago but given my recent post on Lifecycle I thought I'd revisit it. I will assume the reader is entirely familiar with the concepts of lifecycle and commoditisation.

In figure 1, I've taken part of the lifecycle curve and modeled onto it a differential benefit curve (differential value - cost of implementation). This latter curve shows how the benefit of an activity changes as it evolves from its early innovation (where it is a strain on company resources) to a late product stage where the activity is ubiquitous and of insignificant differential value between competitors.

Figure 1 - Lifecycle & Differential Benefit (click on image for higher resolution)

When a new activity appears, you often get whitepapers written about it. These are generally done at a time when the activity is showing a highly positive differential benefit. Obviously there is a delay between the collection of data, the publication of the whitepaper, a user reading the whitepaper, the decision to do something and then implementation of the activity.

By the time an activity is implemented, the actual differential benefit may be vastly different. This creates a delta for expectation i.e. a difference from what we thought we would get and what we got. Figure 2 provides a graphical notation of this.

Figure 2 - Delta in Expectation (click on image for higher resolution)

Modeling this delta provides the expectation curve shown in figure 3. There are a complex set of assumptions and conditions around this, however for the purpose of this post the curve is generally valid. What the curve shows is the early stages start with low expectations (but possibly high hopes), expectations are then quickly exceeded and continuously rise to reach a plateau after which expectations rapidly become unfulfilled leading to a trough of disillusionment before eventually leveling.

Figure 3 - Expectation Curve (click on image for higher resolution)

This all sounds strangely familiar and so it should. The expectation curve quite neatly maps to Gartner's hype cycle - see figure 4. So, we have a potential basis for explaining the underlying forces behind the hype cycle.

Figure 4 - Hype Cycle & Expectation Curve (click on image for higher resolution)

Does our new underlying basis of the hype cycle shed any new light on the subject? Well, yes. However before I show this, I'd like to examine the final stages of lifecycle.

The evolution of an activity from products to utility services invokes its own expectation curve not through differential value (the creation of a new activity) but operational efficiency (a more efficient means of providing an existing activity).

In figure 5, I've provided the later stages of lifecycle including the transition from products to utility services and modeled an operational benefit curve (operational efficiencies over competitors - cost) of a transition to utility services.

Figure 5 - Lifecycle and Operational Benefit (click on image for higher resolution)

[NB. this benefit curve is the same shape as the earlier differential curve being derived from the benefit created over competitors, number of competitors exploiting the change and a changing cost of implementation due to maturity. Hence the transition to utility services starts with a period of investment, a rapid benefit over competitors gained by those creating or exploiting such services and then a decline in benefit over competitors as more companies switch to a utility model.]

The reason why I mention this, is that whilst Cloud Computing is all about volume operations for ubiquitous and well defined activities (i.e. use of computer resources in business) and is hence all about commodities, this transition will create a similar expectation curve around operational efficiency in much the same way that a genuine innovation creates an expectation curve around differential value. This is shown in figure 6, and the result is the same delta in expectation curve shown beforehand.

Figure 6 - Delta in Expectation (click on image for higher resolution)

Gartner's hype cycle tends to show both innovative activities and transitional effects on the same graph. We should be careful to distinguish between operational efficiency (doing the same thing better) and differential value (doing a new thing) lest we start to confuse Cloud Computing with Innovation.

Hence, in the following hype cycle I've highlighted several activities, including :-

cloud computing: more efficient provision of the existing activity of "using computer resources in business"
social network analysis: a relatively new activity and a potential differential

Figure 7 - Expectation and Hype Cycle (click on image for higher resolution)

Since we started with differential value or operational efficiency, we can now map value zones onto the hype cycle. I've done this below.

Figure 8 - Hype Cycle & Value Zones (click on image for higher resolution)

What this suggests is the early stages of the hype cycle has an increasing benefit. In the trough of disillusionment, there is still benefit to be gained but it's diminishing. However, in the slope of enlightenment and the plateau of productivity there is little or no differential or operational benefit over competitors since everyone else is doing it.

The lesson of this story has been known in military circles for a long time. An imperfect plan executed today is better than a perfect plan executed tomorrow i.e. if you wait until the activity can be easily and effectively implemented (the plateau of productivity), it'll provide little competitive benefit to you.

Fortune favours the brave.

[A final few comments]

To generate the expectation curve I had to create a model over time. This required lots of assumptions because the evolution (lifecycle) curve does not have a time axis (i.e. you can't predict when something will evolve). There are hence a couple of points I'd like to make clear.

You can't simply overlay the expectation curves of different activities on top of each other - i.e. the axis of time is different (some are stretched, some are shortened). Gartner's curve doesn't define its time axis and we can therefore assume they're referring to a general shape which appears over an undetermined length of time.
The Gartner curve specifically refers to the technology trigger. We can assume this is when the technology starts to spread and ignores any early stage effects (invention etc).
If the Gartner curve was based upon the measurement of some physical property, it would be possible to reverse the process i.e. from Gartner curve to expectation curve to evolution lifecyle and accurately state where an activity was along the uncertainty axis. By very definition this is impossible. I can currently only state where something was in the past once it has become a commodity. Hence I have to conclude that Gartner's curve is not based upon some external measurement of physical property but instead it is more likely by a process of expert review (i.e. averaging where forecasters think something is on the curve).
The expectation curve matches the Gartner curve in certain circumstances. I cannot conclude the Gartner curve is valid in all circumstance but I can approximate the value zones where the curve does match.

Pioneers, Town Planners and those missing Settlers.

2011-02-11T16:21:00.015Z

All business activities evolve, they share a fairly common lifecycle described in the following diagram. From innovation, to custom built examples, to productisation (including the appearance of rental services) and finally to commodity (including utility services). I've also added some reference points for CRM to illustrate the change.

Figure 1 - Lifecycle (click on image for higher resolution)

As those activities evolve, their properties change from a chaotic to a linear extreme. In the chaotic stage, the activity :-

deviates from what has existed before and is a novel practice.
is dynamic and constantly changing.
is rare and poorly understood.
has high levels of uncertainty and it is not possible to predict future outcomes.
has no market data, competitor analysis or well understood trends.
has characteristics which emerge as we learn about it.
is strongly affected by serendipity, chance encounters and discovery.
is a potential source of future worth, differential and hence competitive advantage.
is a gamble

By the linear stage, that same activity has evolved and:-

is mature and rarely changes.
is standardised with a wealth of best practice.
is commonplace and well understood.
has a high degree of certainty and known impacts.
has an abundance of market data, competitor analysis and trends are well known.
has well defined characteristics.
has well defined procedures and plans for implementation.
is a cost of doing business with little or no differential advantage except through operational efficiencies.
is a known quantity.

Now all businesses consist of a mass of activities, each of which may be at different stages of their lifecycle. By plotting the frequency of activities at different stages, a profile for an organisation or an industry can be created. This is shown in the figure below, to which the chaotic, linear and in-between stage of transition has been added.

Figure 2 - Profile (click on image for higher resolution)

The techniques which you use to manage each of the phases of profile (chaotic, transition, linear) are entirely different because the fundamental characteristics are different. Which is why no one size fits all approach to management exists. For example, agile development approaches are ideal for the innovation (chaotic) and early transition phases but are superseded by more structured approaches such as six sigma in the late transition and commodity (linear) stages. You can't apply one size fits all without either hampering innovation or impacting efficiency.

In many areas of management, this creates a constant yo-yo between one approach and another such as : agile vs six sigma, networked vs hierarchical, push vs pull. The answer is invariably you need a balance of both. The trick is to learn when to use each.

Given all this, here are my questions :-

1. Since lifecycle is constant and the properties of activities change as they evolve through their lifecycle, why do we organise ourselves around type of activities (e.g. IT, Finance, Operations) especially as a "typed" approach leads to outsourcing of inappropriate activities, misapplied techniques & alignment issues between groups?

2. Why don't we organise ourselves instead by lifecycle with specialist groups managing each stage of lifecyle regardless of the type i.e. an organisation based upon Pioneers, Settlers and Town Planners?

3. Most companies have Research & Development groups (equivalent to Pioneers) and common or shared service groups (equivalent to Town Planners) but Settlers seem to be invisible. Why is this? Who manages the transition from innovation to commodity in your organisation?

Top tips for start-ups

2011-01-16T19:24:00.003Z

By popular request, here are Andrey Markov's top tips for startups, as derived from many leading VCs' posts on what makes a successful startup.

P.S. Before taking this seriously, please read the note at the end.

=== Start Text ===

Top Tips for 99% at least of small startups.

When seeking investment make the founder Geraldine Brooks.
You avoid firing people. But when you're not qualified, the end is the most important thing.
Embrace change. If you are offering to investors hold you pick the MP3 player space, all because you better be afraid as investors are lots of weakness.
Launch into something. Go ahead with this thing. You need to Google, and see the search terms accordingly.  
Choose a way everyone knows what's the bat. Have the agility to further understand your team. Know when you need in three years and good long staff meetings that there is now a market.
Become an expert on revenue. It helps to deploy? Understand your product just as you reach them, and maybe also their business plan.
Get a stunning logo or generate feedback; that means engaging a small business advisors. Let your audience in the company, acknowledge them, but avoid defensive responses. Convince the angels of a sign of myself, will you be “Can everyone win this business?"
Good isn't working the performance. "There are the ship keeps moving forward. Otherwise, you'll soon be abroad when the good are offering to clear."
This may make it a Fortune 500 company, and former colleague told me on it. In most budding entrepreneurs, you can. Make the business school code for startup culture: a friend of your team, and cannot stop less than "I have an airline".

In summary, you get experience and torn jeans as you integrate yourself - they needed helping, say, but the early days we didn't tell you so.

=== End Text ===

The real shame of this, is it actually makes more sense than some of the stuff I get to read. Special thanks to Doctor Nerve's Markov Text Generator and yes, this is the last one.

A full and workable definition for cloud?

2011-01-16T12:56:00.010Z

Definition provided courtesy of Andrey Markov and the texts of many prominent cloud figures.

P.S. Before taking this seriously, please read the note at the end.

=== Start Text ===

Cloud data centers have lower spend per data centers. The authors lay out peaks and reassigned according to recognize that facilitate incremental improvement and four deployment models. Any CIO will use software to take full advantage and ultimately overrides initial concerns (e.g., mission, security holes, high compute task implementation).

When computing offer enough economic advantages will come to achieve higher level of use and smooth out peaks and elastically provisioned, in terms of IT being service oriented (as the paper "The Economics of service models") and reliability will undoubtedly be able to be increased significantly, thereby reducing operational costs caused by heterogeneous thin client interface or thick client platforms (e.g. networks) and can be recognized, and cost of adoption despite initial reservations, resulting in a private cloud with different physical characteristics and reduces cost of the same subject.

"It's the entire industry group and reduces cost advantage include operating systems, storage, applications, and applications".

The basis for that underlie cloud computing? There is likely to a third party and possibly application architecture design pattern learning present themselves as examples. Their point is nothing new (application capabilities, fragility, security spend per data center). The cloud bursting for a significant input to upgrade. We put up as needed automatically without doubt.

Beyond the consumer is in part, because of power management is composed of the TCO (per server, based on the cloud with fragility, security and compliance considerations). It may be unlimited and reliability will adopt technologies despite technical concerns is to vastly increase the rapid innovation we've grown accustomed to. To this is remarkable.

In concluding the authors then ventures into what types of disruptions, as a much easier to use and supports a cloud computing initiative, or service models (and services) that of scale available over time. Their point is available to yourself to be its article; nevertheless, the consumer does cloud computing in the TCO factor compared to the original UC Berkeley Cloud Platform as noted, this question, the economic advantage, organizations will gradually fade, thus resulting in terms of service (e.g. host firewalls).

Deployment Models: Private cloud. The basis for the deployed applications, and accessed through the possible exception of the provided resources include operating systems or acquired applications. Programming languages are reported providing transparency for use of 1000 hours, more compute-intensive tasks will result from (titled "Data center is identified, negotiate extremely low rates than that"). I feel this white paper will be summed up as well. Instead the provider’s applications running on to this white paper is nothing but application hosting environment configurations. Cloud Software as a private variant runs a single document and helps shape the same subject:

"It's the implications of configurable computing economics".

Essential Characteristics: On-demand self-service. A copy of business units, or live with cloud infrastructure is operated solely for 1000 servers for a thin or getting ready to be very large volumes of the ratio of public cloud computing. I would be recognized, and may be able to upgrade. We put up with minimal management is composed of scale advantage.

Supply-side savings: Cloud computing capabilities, such as server data center buildouts, the scale of these cost = increased use. Elasticity of the cloud infrastructure is made available over operating systems, storage, processing, memory, network bandwidth, and lay out peaks and consumer of the underlying economics that economic advantage, organizations and concerns is going to deploy onto the consumer with cloud infrastructure.

In end, the cloud computing does not manage or getting ready to which cloud infrastructure is.

=== End Text ===

The real shame of this, is it actually makes more sense than some of the stuff I get to read. Thanks to @b6n for the suggestion of running a set of popular cloud posts through a Markov Text Generator.

My top 10 influential thinkers in cloud ...

2011-01-05T22:19:00.009Z

I've never been a great fan of top ten lists, however since it appears fashionable I thought I'd have a go. So, here are my top 10 most influential thinkers to the cloud in order of priority:

Douglas Parkhill: For predicting the entire field and writing the exceptional book "The Challenge of the Computer Utility" (1966)
Joseph Schumpeter: For providing a basic economic framework which explains why cloud computing should enable further innovation through creative destruction (1942)
Herbert Simon: For providing a basic economic framework which explains why cloud computing should accelerate further innovation through componentisation (1973)
William Stanley Jevons: For outlining why cloud computing won't reduce overall IT expenditure (1865)
Leigh Van Valen: For providing in "a new evolutionary law" a framework to explain why you wouldn't have choice over cloud computing (1973)
Tim O'Reilly: For signposting the cloud computing future with the concept of infoware and highlighting the role of the internet and open source in this (1999)
Everett Rogers: For postulating that diffusion and maturation of a technological innovation results in increased information about the technology and therefore reduces uncertainty about the change. A key cornerstone of the idea of commoditisation (1981)
Paul Strassmann: For demonstrating that there was no correlation between IT spending and business value, hence showing that not all IT is the same and that some was little more than a cost of doing business i.e. it had become more of a commodity (1990s)
Nick Carr: For showing that ubiquity was the key to diminishing strategic value in business and providing the crucial link to explain commoditisation (2003)
John McCarthy: For being the first person to publicly state the idea of utility computing (1961)

Mystic Me 4.0

2011-01-05T14:37:00.012Z

After hitting a 50% prediction rate and fortuitously realising the importance of that, I thought I'd enter the spirit of things again by upping the stakes even further in terms of both specificity and non-obviousness.

As per normal, I'll keep with my insistence on all elements being correct for the prediction to be successful, a dead parrot is a dead parrot in my book. These predictions are what I believe is going to happen but I've pushed them to a level that I can't be certain. The target is to be 50% right (i.e. above that they not either specific or non-obvious enough and vice versa)

So, with no more pining for the Fjords, the Mystic Me 2011 set are :-

Cloud: 2011 will be an interesting year as conventional wisdom within the popular press shifts towards seeing open source architectures dominating the cloud computing space. Cost efficiency arguments around cloud computing will increasingly be replaced with customer innovation stories and the adoption rates of cloud computing will outstrip many early analyst predictions. In particularly pundits will cite AWS as exceeding $1 billion in revenue. With the growth of cloud computing, Enterprise IT will increasingly focus on new value creation, architecture and vendor management techniques and hence there will be increasing mention of terms like supply chain management and new business models based upon outcome. There will be a marked sea change as Platform as a service (PaaS) will overtake Infrastructure as a service (IaaS) as the main buzz of cloud computing. There will also be no let up in the pace of mergers and acquisitions in this industry. Governments will also increasingly become engaged in discussing regulation of the cloud and somewhere, some official will be talking up the idea of licensed cloud operators.
Environment: Total Arctic Ice volume will decline to the lowest level on record with the melting season considered to have extended by several weeks. The UK will suffer another cold winter.
Economy: Inflation, as measured by RPI, will continue to rise however despite this and because of instabilities in the recovery the MPC will hold interest rates low and implement a final last gasp effort for more quantitative easing - principally because they're all barking mad. In the UK, London will experience a property bubble for high value residential property whilst the overall housing market, according to the Halifax House Price Index, will suffer a fall in prices. The double dip will finally arrive in 2011, driven by overexposure of banks to instruments based on sovereign debt during a time when there are increasing market attacks on sovereign debt and a drop in consumer confidence. The FTSE 100 will drop below 3,000 during the year.
Society: As a consequence of austerity measures, general strife and frustration, we will see increasing civil disobedience in many countries. In the UK, I expect to see further protests and increasingly strike action. Despite the necessity to reduce debt, the coalition (in particular the Liberal Party) will continue to wain in popularity polls but despite many pundits predicting an election, the coalition will muddle through.
Technology Business: There are five particular events I expect to see in 2011. First, VMWare will increasingly act as two operational divisions - one focused on infrastructure, the other on platform. Some public pundits will start to question whether one of the units will be sold. Secondly, I expect that whilst CPTN holdings will turn out to be a patent troll, its target is not Android or FOSS specifically but Cloud in general. Thirdly, we should see the examples of companies publicly trading on variability in cloud infrastructure prices through the provision of true brokerage services. Fourthly, the volume of tablet sales will sky rocket with new competitors flooding into the market and the only thing more surprising than the growth of tablets will be the rapid decline of traditional laptops. Lastly, the concept of social searching will become increasingly important with a continuation of the plethora of start-ups providing new ways of ranking, mining and determining social reputation however those pundits discounting the future of Google will get a rude awakening.
Media Technology: In the UK, there will be further high profiled efforts to carve up the Internet. Not only will we see two tiered internet services but also the use of government regulation to introduce censorship based services designed to "protect the most vulnerable". Whilst Paywalls will continue to be the rage, the largest effect will come through the proliferation of devices with on-chip DRM. Many pundits will raise the question whether these devices and the introduction of two tier environments means the Internet can be effectively controlled for the average consumer. Online video will continue to grow exponentially, with YouTube becoming increasingly seen as the future distribution channel of media - i.e. if it ain't on YouTube, did anyone see it?
Manufacturing Business: Printed electronics will have a robust year in the popular press, with pundits talking up the potential for this technology especially when combined with 3D printing. Whilst this is nothing new, there will be a significant increase on the hype around the subject with a couple of high profile articles.
Words to watch for: Consumerization, Shadow IT and Ecosystem are the watch words for 2011 and this is the year in which they'll reach fever pitch. Cloud computing will still cause confusion for many and unfortunately there will be a continuation of marketing efforts to distinguish between enterprise and public cloud.
Social Mobility: Despite increases in tax and a crackdown on tax avoidance causing the usual round of noise about the "wealthy will leave the country" there will be no mass exodus of wealth from the UK.
MISOG's: Despite the occasion, there will be a considerable amount of grumbling over the Royal Wedding and how much coverage it's getting. There will be numerous articles on whether the Royals are still relevant, endless facts on the cost to the economy and assertions that public money would have been better spent elsewhere. Whilst the direct cost of the wedding (ignoring overall economic impacts) could be paid for several times over out of the tax efficiency measures that Philip Green (our cost Czar) has employed or the huge profits institutions have made on quantitative easing, unfortunately no private company will step upto the plate and offer to pay the bill. Someone, somewhere will write an article about how the cost of giving everyone an extra days holiday could stall the UK recovery.

Review of Mystic Me 3.0

2011-01-01T15:52:00.006Z

I've not posted for a short while, so I thought I'd start with a round-up on how I did with last year's predictions before providing some more cowardly custard predictions for 2011.

I thought that 2010 was a bad year for me in the prediction stakes because I hit a 50% rate. However, on reflection this doesn't seem to be the case. The issue with prediction is uncertainty and value i.e the level of detail and non-obviousness within the prediction. It is trivially easy to define a broad set of predictions on well established trends that means you're almost always 100% right but then they're not much use.

Taking the case of specificity, a prediction of a single number coming up in any of the 52 weeks of the lottery is very high whereas the odds of predicting what the entire set of lottery numbers are on a specific week is very low. Equally, predicting continuation of an established and understood trend (e.g. "I predict the sun will come up tomorrow") is of little value. Not all predictions are equal.

So how do you strike the balance? Each year, I've upped the stakes on both the specificity and non-obviousness of the predictions hence increasing the probability of failure. However in doing so I've failed to understand that the target should not be to achieve 100% success but instead 50% as an optimal goal i.e. a level of specificity and non-obviousness that means you're entirely right half the time. The future is uncertain and predictions should reflect this otherwise we're not pushing the prediction hard enough.

With that said, here's how I did :-

The number of mergers & acquisitions in the cloud computing and open source industries will reach fever pitch, surpassing previous years: Well, that happened. It was a fairly insane ride in the world of M&A from Novell, 3Tera, CloudKick, Makara to Heroku and a host of others. Score:1
The first examples of people trading on variability in cloud infrastructure prices and the early formation of brokerage concepts will appear: Whilst examples of brokerage concepts appeared, companies started to use reserved, spot and on-demand instances to reduce costs and we even had announcements regarding spot markets, there were no clear examples of people publicly trading on variability. Score: 0
There will be no let-up in end user confusion surrounding cloud computing: Whether it's surveys of Canadian Executives or Technology Resellers, confusion over cloud continues unabated. Score: 1
Despite many predicting the death of the book, paperbacks will have a surprisingly good year: Despite the doomsayers and the rise of kindle, published figures show that paperbacks have held up remarkably well with only modest declines in volume. Score: 1
RPI in the UK will rise sharply and the FTSE 100 will drop below 3,000 during the year. Judging on past performance, the MPC will keep interest rates low because they're barking mad: Whilst RPI rose sharply and the MPC held interest rates level, the double dip has been somewhat delayed by financial engineering. So, whilst I still hold to this prediction, any major drop has been delayed until 2011. Score: 0.
Under howls of protest, banks will be given more taxpayers cash. This will be despite being bailed out, given free cash through quantitative easing and then splashing lots of dosh on bonuses: Well, we have the bonuses, healthy rounds of free cash aka quantitative easing courtesy of the USA but with no double dip and the bail-out of the credit default swap bonanza that was building around sovereign debt, any effect has been delayed until next year. Score: 0
House prices will continue to drop in the UK: Using the same metrics as last year, house prices in the UK (except in London) have dropped. Certainly, the lack of the double dip has kept prices higher. Score: 1
Summer ice disappearance exceeding the worst predictions of current climate models: Artic summer sea ice volume was 4,000 km^3, by far the worst figure on record. Score: 1
There will be legal attempts to claim and quantify ownership of social networks as company IP: Actually, this had already happened before the prediction was made, so that has to be a score : 0 for not checking properly.
The new Doctor Who will be pants and the attempts to spice it up and make it more gritty will look rather sad: Whilst the attempts to spice up and make Dr Who a bit more gritty are public record, I have to say that Matt Smith is excellent in the role. Score: 0

Overall, 5/10. I had originally though this was a "could do better" score instead I realise that this is exactly what I should aiming for.

All in a word.

2010-11-19T14:44:00.007Z

In my previous post, I provided a more fully fledged version of the lifecycle curve that I use to discuss how activities change. I've spoken about this for many years but I thought I spend a little time focusing on a few nuances.

Today, I'll talk about the *aaS misconception - a pet hate of mine. The figure below shows the evolution of infrastructure through different stages. [The stages are outlined in the previous post]

Figure 1 - Lifecycle (click on image for higher resolution)

I'll note that service bureau's started back in the 1960s and we have a rich history of hosting companies which date well before the current "cloud" phenomenon. This causes a great deal of confusion over who and who isn't providing cloud.

The problem is the use of the *aaS terms such as Infrastructure as a Service. Infrastructure clouds aren't just about Infrastructure as a Service, they're about Infrastructure as a Utility Service.

Much of the confusion has been caused by the great renaming of utility computing to cloud, which is why I'm fairly consistent on the need to return to Parkhill's view of the world (Challenge of the Computer Utility, 1966).

Cloud exists because infrastructure has become ubiquitous and well defined enough to support the volume operations needed for provision of a commodity through utility services. The commodity part of the equation is vital to understanding what is happening and it provides the distinction between a VDC (virtual data centre) and cloud environments.

If you're building an infrastructure cloud (whether public or private) then I'll assume you've got multi-tenancy, APIs for creating instances, utility billing and you are probably using some form of virtualisation. Now, if this is the case then you're part of the way there, so go check out your data centre.

IF :-

your data centre is full of racks or containers each with volumes of highly commoditised servers
you've stripped out almost all physical redundancy because frankly it's too expensive and only exists because of legacy architectural principles due to the high MTTR for replacement of equipment
you're working on the principle of volume operations and provision of standardised "good enough" components with defined sizes of virtual servers
the environment is heavily automated
you're working hard to drive even greater standardisation and cost efficiencies
you don't know where applications are running in your data centre and you don't care.
you don't care if a single server dies

... then you're treating infrastructure like a commodity and you're running a cloud.

The economies of scale you can make with your cloud will vary according to size, this is something you've come to accept. But when dealing with scale you should be looking at :-

operating not on the basis of servers but of racks or containers i.e. when enough of a rack is dead you pull it out and replace it with a new one
your TCO (incl hardware/software/people/power/building ...) for providing a standard virtual server is probably somewhere between $200 - $400 per annum and you're trying to make it less.

Obviously, you might make compromises for reasons of short term educational barriers (i.e. to encourage adoption). Examples include: you might want the ability to know where an application is running or to move an application from one server to another or you might even have a highly resilient section to cope with many legacy systems that have developed with old architectural principles such as Scale-up and N+1. Whilst these are valuable short term measures and there will be many niche markets carved out based upon such capabilities, they incur costs and ultimately aren't needed.

Cost and variability are what you want to drive out of the system ... that's the whole point about a utility.

Anyway, rant over until next week.

IT Extremists

2010-11-14T16:08:00.013Z

The problem with any transition is that inevitably you end up with extremists, cloud computing and IT are no exception. I thought I'd say a few words on the subject.

I'll start with highlighting some points regarding the curve which I use to describe the underlying transition (evolution) behind cloud. I'm not going to simplify the graph quite as much as I normally do but then I'll assume it's not the first time readers have seen this.

Figure 1 - Lifecycle (click on image for higher resolution)

The points I'll highlight are :-

IT isn't one thing it's a mass of activities (the blue crosses)
All activities are undergoing evolution (commonly known as commoditisation) from innovation to commodity.
As activities shift towards more of a commodity, the value is in the service and not the bits. Hence the use open source has naturally advantages particularly in provision of a marketplace of service providers.
Commoditisation of an activity not only enables innovation of new activities (creative destruction), it can accelerate the rate of innovation (componentisation) of higher order systems and even accelerate the process of evolution of all activities (increase communication, participation etc).
Commoditisation of an activity can result in increased consumption of that activity through price elasticity, long tail of unmet demand, increased agility and co-evolution of new industries. These are the principle causes of Jevons' paradox.
As an activity evolves between different stages risks occur including disruption (including previous relationships, political capital & investment), transition (including confusion, governance & trust) and outsourcing risks (including suitability, loss of strategic control and lack of pricing competition.
Benefits of the evolution of an activity are standard and include increased efficiencies (including economies of scale, balancing of heterogeneous demand etc), ability of user to focus on core activities, increased rates of agility and tighter linking between expenditure and consumption.
Within a competitive ecosystem, adoption of a more evolved model creates pressure for others to adopt (Red Queen Hypothesis).
The process of evolution is itself driven by end user and supplier competition.
The general properties of an activity changes as it evolves from innovation (i.e. dynamic, deviates, uncertain, source of potential advantage, differential) to more of a commodity (i.e.repeated, standard, defined, operational efficiency, cost of doing business).

The above is a summary of some of the effects, however I'll use this to demonstrate the extremist views that appear in our IT field.

Private vs Public Cloud: in all other industries which have undergone this transition, a hybrid form (i.e. public + private) appeared and then the balance between the two extremes shifted towards more public provision as marketplaces developed. Whilst private provision didn't achieve (in general) the efficiencies of public provision, it can be used to mitigate transitional and outsourcing risks. Cloud computing is no exception, hybrid forms will appear purely for the reasons of balancing benefits vs risks and over time the balance between private and public will shift towards public provision as marketplaces form. Beware ideologists saying cloud will develop as just one or the other, history is not on their side

Commoditisation vs Innovation: the beauty of commoditisation is that it enables and accelerates the rate of innovation of higher order systems. The development of commodity provision of electricity resulted in an explosion of innovation in things which consumed electricity. This process is behind our amazing technological progress over the last two hundred years. Beware those who say commoditisation will stifle innovation, history says the reverse.

IT is becoming a commodity vs IT isn't becoming a commodity: IT isn't one thing, it's a mass of activities. Some of those activities are becoming a commodity and new activities (i.e. innovations) are appearing all the time. Beware those describing the future of IT as though it's one thing.

Open Source vs Proprietary : each technique has a domain in which it has certain advantages. Open source has a peculiarly powerful advantage in accelerating the evolution of an activity towards being a commodity, a domain where open source has natural strengths. The two approaches are not mutually exclusive i.e. both can be used. However, as activities become provided through utility services, the economics of the product world doesn't apply i.e. most of the wealthy service companies in the future will be primarily using open source and happily buying up open source and proprietary groups. This is diametrically opposed to the current product world where proprietary product groups buy up open source companies. Beware the open source vs proprietary viewpoint and the application of old product ideas to the future.

I could go on all night and pick on a mass of subjects including Agile vs Six Sigma, Networked vs Hiearchical, Push vs Pull, Dynamic vs Linear ... but I won't. I'll just say that in general where there exists two opposite extremes, the answer normally involves a bit of both.

A run on your cloud?

2010-09-21T22:19:00.014+01:00

When I use a bank, I'm fully aware that the statement I receive is just a set of digits outlining an agreement of how much money I have or owe. In the case of savings, this doesn't mean the bank has my money in a vault somewhere as in all likelihood it's been lent out or used elsewhere. The system works because a certain amount of reserve is kept in order to cover financial transactions and an assumption is made that most of my money will stay put.

Of course, as soon as large numbers of people try to get their money out, it causes a run on the bank and we discover just how little the reserves are. Fortunately, in the UK we have an FSA scheme to guarantee a minimum amount that will be returned.

So, what's this got to do with cloud? Well, cloud (as with banking) works on a utility model, though in the case of banking we get paid on both the amount we consume and provide (i.e interest) and in the cloud world we normally only have the option to consume.

In the case of infrastructure service providers, there are no standard units (i.e. there is no common cloud currency) but instead each provider offers it own range of units. Hence if I rent a thousand computer resource units, those units are defined by that provider as offering a certain amount of storage and CPU for a given level of quality at specified rate (often an hourly fee).

As with any utility there is no guarantee that when I want more, the provider is willing to offer this or has the capacity to do so. This is why the claims of infinite availability are no more than an illusion.

However, hidden in the depths of this is a problem with transparency which could cause a run on your cloud in much the same way that Credit Default Swaps hit many financial institutions as debt exceeded our capacity to service it.

When I rent a compute resource unit from a provider, I'm working on the assumption that what I'm getting is that compute resource unit and not some part of it. For example, if I'm renting on an hourly basis a 1Ghz core with 100Gb storage and 2Gb memory - I'm expecting exactly that.

However, I might not use the whole of this compute resource. This offers the service provider, if they were inclined, an opportunity to sell the excess to another user. In this way, a service provider running on a utility basis could be actively selling 200 of their self defined compute units to customers whilst it only has the capacity to provide for 100 of those units when fully used. This is quaintly given terms like improving utilisation or overbooking or oversubscription but fundamentally it's all about maximising service provider margin.

The problem occurs when everyone tries to use their compute resources fully with an overbooked provider, just like everyone trying to get their money out of a bank. The provider is unable to meet its obligations and partially collapses. The likely effect will be compute units being vastly below their specification or some units which have been sold are thrown off the service to make up for the shortfall (i.e. customers are bumped).

It's worth remembering that a key part of cloud computing is a componentisation effect which is likely to lead to massively increased usage of computer infrastructure in ever more ephemeral infrastructures and as a result our dependency on this commodity provision will increase. It's all worth remembering that black swan events, like bank runs do occur.

If one overbooked provider collapses, then this is likely to create increased strain on other providers as users seek alternative sources of computer resource. Due to such an event and unexpected demand, this might lead to a temporary condition where some providers are not able to hand out additional capacity (i.e. new compute units) - the banking equivalent of closing the doors or localised brown-outs in the electricity industry.

However, people being people will tend to maximise the use of what they already have. Hence, if I'm renting 100 units with one provider who is collapsing, 100 units with another who isn't and a situation where many providers are closing their doors temporarily, then I'll tend to double up the workload where possible on my fully working 100 units (i.e where I believe I have spare capacity).

Unfortunately, I won't be the only one doing this and if that provider has overbooked then it'll collapse to some degree. The net effect is a potential cascade failure.

Now, this failure would not be the result of poor utility planning but instead the overbooking and hence overselling of capacity which does not exist, in much the same way that debt was sold beyond our capacity to service it. The providers have no way of predicting black swan events, nor can they estimate the uncertainty with user consumption (users, however, are more capable of predicting there own likely demands).

There are several solutions to this, however all require clear transparency on the level of overbooking. In the case of Amazon, Werner has made a clear statement that they don't overbook and sell your unused capacity i.e. you get exactly what you paid for.

Rackspace also states that they offer guaranteed and reserved levels of CPU, RAM and Storage with no over subscription (i.e. overbooking).

In this case of VMWare's vCloud Director, then according to James Watters they provide a mechanism for buying a hard reservation from a provider (i.e. a defined unit), with any over commitment being done by the user and under their control.

When it comes to choosing an infrastructure cloud provider, I can only recommend that you first start by asking them what units of compute resource they sell? Then afterwords, ask them whether you actually get that unit or merely a capacity for such depending upon what others are doing? In short, does a compute unit of 1Ghz core with 100Gb storage and 2Gb memory actually mean that or could it mean a lot less?

It's worth knowing exactly what you're getting for your buck.

The bets ...

2010-08-28T19:14:00.010+01:00

Every now and then I make predictions about the future, sometimes I even make a bet. I thought I'd write down some of the outstanding bets I have, obviously leaving out the counterparty & how much it is for.

[2006] : The Open Source Bet.
By the end of 2015, it will be accepted wisdom that open source provides the only viable mechanism of creating competitive markets in the utility computing world. It will dominate this space and be considered the norm.

[2006] : The JS Bet.
By the end of 2015, JavaScript will be seen as a rapidly growing enterprise language for developing entire applications, both client and server side.

[2007] : The Broker Bet.
By the end of 2013, we will see adverts for computer resource brokers.

[2008] : The Three Step Bet.
By the end of 2013, Oracle would have bought Sun in order to get into the cloud space, RedHat would have bought Novell and Microsoft will have bought Canonical.

[2009] : The Escape Button Bet.
By the end of 2014, VMWare will have divided into two companies, one focused on IaaS technology and the other focused on PaaS technology. The IaaS group will have been sold off to another company.

[2010] : The Rackspace Bet.
By Aug 2013, Rackspace will be providing EC2/S3 APIs alongside its own.

Added Friday 25th March 2011

Rather than creating new lists, I thought I'd extend this list.

[2011] : The Apple Bet (@GeorgeReese & @mthiele10)
By the end of 2017, Apple will have or will be in the process of filing for protection under chapter 11 of the U.S. bankruptcy code.

Arguably, the best cloud conference in the world?

2010-08-18T10:16:00.009+01:00

For those of you who missed the OSCON Cloud Summit, I've put together a list of the videos and speakers. Obviously this doesn't recreate the event, which was an absolute blast, but at least it'll give you a flavour of what was missed.

Welcome to Cloud Summit [Video 14:28]
Very light introduction into cloud computing with an introduction to the speakers and the conference itself. This section is only really relevant for laying out the conference, so can easily be skipped.
With John Willis (@botchagalupe) of opscode and myself (@swardley) of Leading Edge Forum.

Scene Setting
In these opening sessions we looked at some of the practical issues that cloud creates.

Is the Enterprise Ready for the Cloud? [Video 16:39]
This session examines the challenges that face enterprises in adopting cloud computing. Is it just a technology problem or are there management considerations? Are enterprises adopted cloud, is the cloud ready for them and are they ready for it?
With Mark Masterson (@mastermark) of CSC.

Security, Identity – Back to the Drawing Board? [Video 25:12]
Is much of the cloud security debate simply FUD or are there some real consequences of this change?
With Subra Kumaraswamy(@subrak) of Ebay.

Cloudy Operations [Video 22:10]
In the cloud world new paradigms and memes are appearing :- the rise of the “DevOps”, “Infrastructure == Code” and “Design for Failure”. Given that cloud is fundamentally about volume operations of a commoditized activity, operations become a key battleground for competitive efficiency. Automation and orchestration appear key areas for the future development of the cloud. We review current thinking and who is leading this change.
With John Willis (@botchagalupe) of opscode.

The Cloud Myths, Schemes and Dirty Little Secrets [Video 17:38]
The cloud is surrounded by many claims but how many of these stand up to scrutiny. How many are based on fact or are simply wishful thinking? Is cloud computing green, will it save you money, will it lead to faster rates of innovation? We explore this subject and look at the dirty little secrets that no-one wants to tell you.
With Patrick Kerpan (@pjktech) of CohesiveFT.

Curing Addiction is Easier [Video 18:41]
Since Douglas Parkhill first introduced us to the idea of competitive markets of compute utilities back in the 1960s, the question has always been when would this occur? However, is a competitive marketplace in the interests of everyone and do providers want easy switching? We examine the issue of standards and portability in the cloud.
With Stephen O’Grady (@sogrady) of Redmonk.

Future Setting
In this section we heard from leading visionaries on the trends they see occurring in the cloud and the connection and relationships to other changes in our industry.

The Future of the Cloud [Video 29:00]
Cloud seems to be happening now but where is it going and where are we heading?
With J.P. Rangaswami (@jobsworth) of BT.

Cloud, E2.0 – Joining the Dots [Video 30:04]
Is cloud just an isolated phenomenon, or is it connected to many of the other changes in our industries.
With Dion Hinchcliffe (@dhinchcliffe) of Dachis.

The Questions
The next section was a Trial by Jury where we examined some of the key questions around cloud and open source.

What We Need are Standards in the Cloud [Video 45:17]
We put this question to the test, with prosecution Benjamin Black (@b6n) of FastIP, defence Sam Johnston (@samj) of Google and trial by a Jury of John Willis, Mark Masterson, Patrick Kerpan & Stephen O’Grady

Are Open APIs Enough to Prevent Lock-in? [Video 43:21]
We put this question to the test, with prosecution James Duncan (@jamesaduncan) of Joyent, defence George Reese (@georgereese) of Enstratus and trial by a Jury of John Willis, Mark Masterson, Patrick Kerpan & Stephen O’Grady

The Debates
Following the introductory sessions, the conference focused on two major debates. The first of these covered the “cloud computing and open source question”. To introduce the subject and the panelists, there were a number of short talks before the panel debates the impact of open source to cloud and vice versa.

The Journey So Far [Video 10:59]
An overview of how “cloud” has changed in the last five years.
With James Urquhart (@jamesurquhart) of CISCO.

Cloud and Open Source – A Natural Fit or Mortal Enemies? [Video 8:44]
Does open source matter in the cloud? Are they complimentary or antagonistic?
With Marten Mickos (@martenmickos) of Eucalyptus.

Cloudy Futures? The Role of Open Source in Creating Competitive Markets [Video 8:43]
How will open source help create competitive markets? Do “bits” have value in the future and will there be a place for proprietary technology?
With Rick Clark (@dendrobates) of OpenStack.

The Future of Open Source [Video 9:34]
What will cloud mean to open source development and to linux distributions. Will anyone care about the distro anymore?
With Neil Levine (@neilwlevine) of Canonical.

The Debate – Open Source and the Cloud  [Video 36:24]
Our panel of experts examined the relationship between open source and cloud computing.
With Rick Clark, Neil Levine, Marten Mickos & James Urquhart

The Future Panel followed the same format with first an introduction to the experts who will debate where cloud is going to take us.

The Government and Cloud [Video 10:27]
The role of cloud computing in government IT – an introduction to the large G-Cloud and App Store project under way in the UK; what the UK public sector hopes to gain from a cloud approach, an overview of the proposed technical architecture, and how to deliver the benefits of cloud while still meeting government’s stringent security requirements.
With Kate Craig-Wood (@memset_kate) of Memset.

Infoware + 10 Years [Video 10:38]
Ten years after Tim created the term infoware, how have things turned out and what is the cloud’s role in this?
With Tim O'Reilly (@timoreilly) of O'Reilly Media.

The Debate – A Cloudy Future or Can We See Trends? [Video 50:12]
The panel of experts examine what’s next for cloud computing, what trends can they forsee.
With Kate Craig-Wood, Dion Hinchcliffe, Tim O’Reilly & JP Rangaswami

So, why "arguably the best cloud conference in the world?"

As a general conference on cloud, then the standard and quality of the speakers was outstanding. The speakers made the conference, they gave their time freely and were selected from a wide group of opinion leaders in this space. There was no vendor pitches, no paid for conference speaking slots and hence the discussion was frank and open. The audience themselves responded marvelously with a range of demanding questions.

It is almost impossible to pick a best talk from the conference because they were all great talks. There are real gems of insight to be found in each and every one and each could easily be the keynote for most conferences. In my opinion, if there is a TED of cloud, then this was it.

Overall, the blend of speakers and audience made it the best cloud conference that I've ever attended (and I've been to 50+). This also made my job as a moderator simple.

I'm very grateful to have been part of this and so my thanks goes to the speakers, the audience, the A/V crew who made life so easy and also Edd Dumbill (@edd), Allison Randal (@allisonrandal), Gina Blaber (@ginablaber) and Shirley Bailes (@shirleybailes) for making it happen.

Finally, huge thanks to Edd and Allison for letting me give a version of my Situation Normal, Everything Must Change talk covering cloud, innovation, commoditisation and my work at LEF.

Islands in the sky

2010-08-04T10:31:00.014+01:00

I'm often asked how will the cloud develop to which I'll answer -"imperfectly, very imperfectly".

I was reminded of this through a long discussion with Benjamin Black, hence I thought I'd write something to explain my general thoughts on the problem. First, let me apologise as this will be a long post. Second, we need to start by recaping some basic concepts about risks. The barriers to adoption in cloud cover three basic forms of risk :-

Disruption Risks : Change to existing business relationships combined with issues around political capital and previous enterprise investment. It's often difficult to let go of that which we have previously invested in.

Transitional Risks: These risks are related to the shift from a world of products to a world of services and they include confusion over the models, trust in the service providers, governance of this service world, transparency from the providers and security of supply. Many of the transitional risks can be mitigated with a hybrid (private + public) cloud approach, a standard supply chain management technique. This approach has been used in many industries which have undergone a similar change, for example in the early decades of power generation it was common to combine public generation with private generators. Even today most data centres mix a variety of public suppliers with backup generators and UPS systems. Fortunately, these transitional risks are relatively short lived.

Outsourcing Risks: These cover lack of pricing competition between the new providers , lack of second sourcing options between providers, loss of strategic control to a specific technology vendor, lock-in and unsuitability of the activity for such service provision (i.e. it's not ubiquitous or well defined enough for such volume operations based service provision). The outsourcing risks can be reduced through the formation of a competitive marketplace of providers with easy switching between them and ideally the option to in-house service provision. The outsourcing risks are long term.

For a competitive market to form, you need easy switching which means portability. The basic ingredients of portability include a choice of providers, access to your code and data from any provider and semantic interoperability between providers i.e. both the origin and destination providers need to understand your code and data in the same way. There is limited value in having access to your code and data if no other provider understands it and operates to provide the same functionality e.g. getting access to your data in salesforce is great but what do you do with it?

In such circumstances, there does exist a weaker form of syntactic interoperability, which means both providers can exchange data but the end result may not function in the same way and your data may not retain its original meaning. Often, this is where we see translation systems to convert from one system to another with the usual abundance of translation and semantic errors.

The ideal situation is therefore semantic interoperability, which generally means a common reference model (i.e. running code) which providers either operate or conform to. Unfortunately, common reference models come with their own risks.

Let us suppose you have a marketplace of providers offering some level of service at a specific level of the computing stack (SPI Model) and these providers operate to a common reference model. The model provides APIs and open data formats, giving you access to your code and data. You therefore have a choice in providers, access to your data and semantic interoperability between them. You have portability. BUT, if that common reference model is owned by a vendor (i.e. it's proprietary code) then that market is not free of constrant but instead controlled by the vendor. All the providers & consumers in that marketplace hand over a significant chunk of strategic control and technology direction to the vendor, who is also able to exert a tax on the market through license fees.

To reduce this loss of strategic control and provide a free market (as in free of constraints), then that common reference model must not be controlled by one party. It has to be open sourced. In such an environment, competition is all about operational efficiency and price vs QoS rather than bits. This makes intuitive sense for a service world, which is why I'm pleased openstack is following that route. Obviously, you'll need different common reference models at different layers of the computing stack. Whilst only one is probably needed for infrastructure, you will need as many as there are competitive application marketplaces (CRM, ERP etc) in the software later of the SPI model.

Before anyone cries the old lie of standardisation hampers innovation, it's worth remembering that utility service provision (which is what cloud is really about) requires volume operations which in turn requires a ubiquitous and well defined activity. Whilst the common reference models certainly won't be perfect in the beginning, they don't need to be, they only have to create "good enough" components (such as a defined virtual machine). They will improve and evolve over time but the real focus of innovation won't be on how good these "good enough" components are but instead what is built with them. This concept, known as componentisation, is prevalent throughout our industrial history and shows one consistent theme - standardisation accelerates innovation.

So everything looks rosy … we'll have the economics benefits of cloud (economies of scale, increased agility, ability to focus on what matters), competitive marketplace based around multiple providers competing on price vs QoS, the options to use providers or install ourselves or to mitigate risks with a hybrid option, "open" API & data formats giving us access to our code and data, open sourced common reference models providing semantic interoperability, "good enough" components for ubiquitous and well defined activities which will cause an acceleration of innovation of new activities based upon these components … and so on.

Think again.

In all likelihood, we're going to end up with islands in the cloud, marketplaces built around specific ways of implementing a ubiquitous and well defined activity. Don't think of "good enough" components but instead a range of different "good enough" components all doing roughly the same thing. Nuts? It is.

Hence, in the infrastructure layer you're likely to see major islands develop around :-

EC2/S3 including the open source implementations such as Open Stack, Eucalyptus and Open Nebula.
vCloud principally provided through VMWare technology.
The openstack APIs, particularly as Rackspace implements this.
a Microsoft infrastructure based environment.

All of these will be providing their own versions of "good enough" units of virtual infrastructure.

Within those islands you'll have multiple service providers, a competitive marketplace with switching between providers and semantic interoperability based upon a common reference model. The open source projects such as OpenStack are likely to form assurance industries (think moody's rating agencies, compliance bodies) to ensure portability between providers by comparison to the common reference model whereas the proprietary technologies are likely to develop certification bodies (e.g. VMWare Express).

Between islands there will be only syntactic interoperability (with exceptions such as OpenStack which will span multiple Islands), which will mean that you'll require translation of systems from one island to another. Whilst management tools will develop (and already have started) to cover multiple islands and translation between them, this process is imperfect and a constant exercise in chasing different APIs and creating a lowest common denominator (as per libcloud). Of course, I wouldn't be surprised if the libcloud folk were hoping that as a community develops around them, then the providers will offer libcloud as a native API. Such command & conquer strategies rarely succeed.

Given this complexity and since there will be multiple service providers within an island, it's likely that consumers will tend to stick within one island.

Of course, these base components will effect the development of higher order layers of the computing stack and you are likely to see increasing divergence between these islands as you move up the stack. Hence, the platform space on the vCloud island will differ from the platform space on the EC2 / S3 island. We will see various efforts to provide common platforms across both, but each will tend towards the lowest common denominator between the islands and never fully exploit the potential of any. Such an approach will generally fail compared to platforms dedicated to that island, especially as each island consists of multiple providers hence overcoming those general outsourcing risks (lack of second sourcing options etc).

So, the future looks like multiple cloud islands, each consisting of many service providers complying to the standard of that island - either vCloud, EC2/S3 or whatever. Increasing divergence in higher order systems (platforms, applications) between the islands and whilst easy switching between providers on an island is straightforward, shifting between islands requires translation. This is not dissimilar to the linux vs windows worlds with applications and platforms tailored to each. The old style of division will just continue with a new set of dividing lines in the cloud. Is that a problem?

Yes, it's huge if you're a customer.

Whilst cloud provides more efficient resources, consumption will go through the roof due to effects such as componentisation, long tail of unmet business demand, co-evolution and increased innovation (Jevons' paradox). Invariably one of the islands will become more price efficient i.e. there is no tax to a technology vendor who collects their annual license and upgrade fee through a drip feed process. It's this increased dependency combined with price variance which will result in operational inefficiencies for one competitor when compared to another who has chosen the more efficient island. The problem for the inefficient competitor will be the translation costs of moving wholesale from one island to another. This is likely to make today's translations look trivial and in all probability will be prohibitive. The inefficient competitor will be forced therefore to compete on a continual disadvantage or attempt to drive the technology vendor to reduce their taxation on the market.

The choices being made today (many are choosing islands based upon existing investment and political choices) will have significant long term impacts and my come to haunt many companies.

It's for these reasons, that I've recommended to anyone getting involved in cloud to look for :-

aggressively commoditised environments with a strong public ecosystem.
signals that multiple providers will exist in the space.
signals that providers in the space are focused on services and not bits.
an open source reference implementation which provides a fully functioning and operating environment.

In my simple world, VMWare is over-engineered and focuses on resilient virtual machines rather than commodity provision. It's ideal for a virtual data centre but we're talking about computing utilities and it also suffers from being a proprietary stack. Many of the other providers offer "open" APIs but as a point of interest APIs can always be reverse engineered for interoperability reasons and hence there is no such thing as "closed" API.

The strongest and most viable island currently resides around EC2 / S3 with the various open source implementations (such as UEC), especially since the introduction of Rackspace & Nasa's service focused openstack effort.

I don't happen to agree with Simon Crosby that VMWare's latest cloud effort Redwood == Deadwood. I agree with his reasoning for why it should be, I agree that they're on shaky grounds in the longer term but unfortunately, I think many companies will go down the Redwood route for reasons of political capital and previous investment. IMHO I'm pretty sure they'll eventually regret that decision.

If you want my recommendation, then at the infrastructure layer get involved with open stack. At the platform layer, we're going to need the same sort of approach. I have high hopes for SSJS (having been part of Zimki all those years back), so something like Joyent's Smart platform would be in the right direction.

Private vs Public clouds

2010-07-30T19:02:00.010+01:00

I thought this argument has been settled a long time ago, seems not. So, once more dear friends I will put on my best impression of a stuck record.

First what is the difference between a public and a private cloud?

A public cloud (the clue is in the name) is open to the public.
A private cloud (the clue is in the name) is private to some set of people.

Naturally, there are shades of grey. For example, the set of people for which a private cloud is private might be one person, a department, a company, a community, a nation or some sort of collection of the above. It is common to use a variety of notations (community, government etc) to distinguish these constraints on use i.e. what subset of people are allowed to use it.

There is another side to this which is your relationship to the provider. It is either :-

external to you and therefore controlled, operated and run by another party.
internal to you which means it is controlled, operated and run by yourself.

Now, once again there are shades of grey because it is perfectly possible for a community of companies to build a shared cloud environment.

Examples of the notation include :-

AWS offers a public cloud which is external to everyone but Amazon.
Eucalyptus offers technology for a company to build a private cloud which is internal to that company.

You could write a list with examples of each but there is little point as no-one uses this notation. Instead in common parlance we tend to use the term public cloud with a single counterpoint of private cloud to mean a cloud where an organisation makes up the subset of private users and the cloud is provided internally to that organisation.

Now we have our bearings on the terms, this leaves a question ...

Why use a private cloud?

A private cloud (using the common meaning) is one that you control and operate. It hence overcomes - or at least creates the illusion of overcoming - many transitional risks such as governance, trust & security of supply. However, it does so at the potential loss of the economies of scale found in public clouds combined with additional costs such as planning, administration and management.

The choice over whether to use one type of cloud or another is always one of benefits vs risks (whether disruption, transition or outsourcing risks). A hybrid cloud strategy simply refers to using a combination of both public and private clouds to maximise benefits for a given appetite of risk.

Naturally, the actual risk can change with a variety of events. For example, the formation of competitive cloud marketplaces with easy switching between multiple providers reduces outsourcing risks (e.g. lack of pricing competition, loss of strategic control, lack of second sourcing options).

For a consumer of cloud services, the ideal scenario is multiple providers of the same service, the option to implement your own environment and no loss of strategic control or dependency to a single vendor.

For this to scenario happen, the technology must be open sourced and hence the technology owners must first realise that in this cloud world value isn't generated through bits of software but instead through services.

In the same way that it took a book company to disrupt the hosting world by offering commoditised infrastructure services, a hosting company is now doing the same to the world of technology vendors through open source. This is just the start and whilst openstack is currently focused on the infrastructure layer, expect it to move up the computing stack in short order.

There are four companies who in my mind exemplify this whole commodity approach - Rackspace, Amazon, Google and Canonical.

I expect all of them to be titans in this space.

OSCON 2010

2010-07-26T03:34:00.020+01:00

I thoroughly enjoyed the OSCON cloud summit and the talk that I gave at OSCON - the audiences were fantastic and the organisation was superb (huge thanks to Edd, Allison and the O'Reilly crew for making this happen).

I'm really proud to have played my small part in this event as the MC for day, along with John Willis.

I haven't yet talked a great deal on my research, but the keynote at OSCON gives a taste of it - so I thought I'd link to it here. Those who know me, also know that this had been a hobby horse of mine over the last decade. It's finally good to spend some focused time on it though of course these ideas are far from new.

A couple of final notes :-

Utility services are just a domain within the commodity phase of an activity's evolution. There are constraints which will prevent a commodity being provided through services. I sometimes plot on the graph a wider "services" stage, however for the sake of simplicity I've left this out.
The stages of lifecycle are approximate only i.e. this is where products appear, this is where utility services generally appear etc.
Multiple activities can be bundled into a single product. For example the iPhone is a combination of different activities from personal communication to digital recorder to web surfing to time keeper to ... the list is quite long. These activities are all evolving and being implemented by others, which forces Apple to focus on two areas :- the bundling of new innovative activities into the iPhone and application innovation through the App Store. The former is expensive and risky. The later requires development of a strong ecosystem, ideally with users being allowed to create and distribute their own applications. The manner in which Apple manages this is less than ideal and they now face severe disruption from Android. As there is also little exploitation of the wider manufacturers' ecosystem, Apple has cornered itself into creating highly costly & risky innovations with weak leveraging. IMHO, they are in trouble and this should become painfully clear in the next five years unless they change.
The ILC model is generally applicable. I picked examples from cloud providers but equally I could have discussed Canonical with Ubuntu. Canonical ruthlessly commoditises activities to provide a stable core and I'd strongly argue that Rackspace & Canonical point to the future direction of IT.
Open source is the natural end state for any activity described by software which is ubiquitous and well defined. This doesn't mean that open source can't be used earlier, of course it can and there are numerous tactical advantages of doing so, along with benefits such as increased collaboration. However, what I am saying is that by the time an activity has reached the commodity phase then only open source makes sense. Those who have been questioning whether "cloud is the death of open source" have a poor understanding as to what is actually happening.
Open core is in general a tactical anomaly. On the one hand, if successful, it will cause widespread distribution (driving an activity towards more of a commodity) and yet it attempts to generate revenue through proprietary elements which is against the natural state that open core is forcing activities towards. A number of companies have used this approach successfully and have even been bought for huge sums by large companies. However, it still remains a tactical anomaly which attempts to achieve both the benefits of open and closed by being both.
The S-Curves I use are not time based. If you follow the evolution of an activity through specific phases of its lifecycle and plot adoption against time, you will derive a set of non-uniform S-Curves for Roger's diffusion of innovation. It's important to realise that the accelerators I mentioned (open source, participation, network effects) along with others I didn't mention (communication mechanisms, co-evolution etc) alter the speed at which an activity evolves. Whilst, this doesn't impact the S-Curves I use, it does compact Roger's curves of more recent innovations when compared to earlier diffusions.
The speed at which an activity moves across the profile graph (i.e. through its lifecycle) depends upon the activity.
None of these ideas are new. The nearest to new is company profile which I've been refining in the last year from earlier work (between '04-'07) and this refinement is simply a formalisation of already existing concepts. If you watched the video and thought, "that's new", then my only advice is be concerned.
On the question of science, the models presented (S-Curve, Profile) are part of a general hypothesis on the evolution of business activities. Whilst data exists, there is neither the volume of evidence nor independent observation to validate beyond this. Furthermore, whilst the models show some usefulness and can be falsified, they are not predictive (and hence this cannot be considered scientific but remains firmly within the field of philosophy). The reason for this is that in order to generate the graphs and avoid asymptotic behaviour, a definition of commodity is required. The consequence of such is that an activity can only be plotted in terms of relative historical position i.e. after it has become a commodity. This means, all positions of activities which have not become a commodity are uncertain (as per one of the axis of the graph) and therefore approximations. The models do not create a crystal ball and the future is one information barrier we can't get past. Even though the new pattens of organisation are testable it should always be remembered that fitness does not guarantee survival.

That's enough for now, I'll expand the topic sometime later.

OpenStack

2010-07-19T05:59:00.011+01:00

There have been many attempts to create open source ecosystems around cloud computing over the last couple of years. Most of them have not fully adopted the largest public ecosystem (being EC2) or been truly open source (instead using an open core model) or they have lacked the experience of large scale cloud operations.

The recent announcement of Open Stack changes this. Entirely open sourced technology for building and running a cloud, supported by an ecosystem of large companies and agencies (including NASA and Rackspace), provision of the EC2 & S3 APIs and the experience of running a large cloud installation.

This is fantastic news.

If you want my view on how this will turn out, well it's rather simple.

OpenStack's move further consolidates the ecosystem around EC2 / S3 which is not only good news for Amazon but also helps propel Rackspace's position as a real thought leader in this space. It's worth noting that the EC2 / S3 API might be supplanted over time, especially as OpenStack builds a marketplace of providers, unless Amazon becomes more open with it.

The icing on the cake will be if Rackspace itself (which will use the OpenStack technology) provides the EC2 / S3 APIs, in which case the growth and consolidation around Rackspace's efforts and any providers of OpenStack will become immense.

This is also surprisingly good news for Eucalyptus if they move towards an entirely (or at least more open approach. In such circumstance, the probability is we're going to end up with a straight forward "clash of the titans" between Eucalyptus and OpenStack to become the Apache of Cloud Computing. Don't be surprised if Eucalyptus even go so far as to adopt some of OpenStack's work. Marten Mickos is an astute businessman and there are many ways they can turn this to their advantage.

However, in general it's not a good time to be any other infrastructure cloud technology vendor, as Simon Crosby makes clear with his "VMWare Redwood = DeadWood" post.

VMWare's position in the infrastructure space is looking unsurprisingly shaky for the future but then they already know of the oncoming disruption, as they made clear with this interview. Why else do you think that VMWare has been busily acquiring into the platform space? RabbitMQ is also increasingly looking like a great purchase for them.

As for RedHat's cloud strategy - they must be feeling increasingly lonely as if no-one wants to invite them to the party. On the other hand, this is good news for Ubuntu, because of both UEC (powered by Eucalyptus) and OpenStacks involvement with the Ubuntu community. Don't be surprised if Ubuntu launches a "powered by openstack" version.

Best of all, it's great for the end users as they will see real choice and further standardisation of a messy industry in the infrastructure space. Of course, the real beauty is that once this happens we can finally start consolidating and standardising the platform space.

From Slime Mold to Neurons

2010-07-13T11:10:00.013+01:00

I wasn't going to write much about clouds, being focused on my new area of research but I could hardly let James' post go unchallenged.

Before I critique the post, I need to go through some basic genetics for those of you who are new to that subject.

DNA is the accepted means of providing genetic instructions used in the development and functioning of all known living organisms. There are exclusions, such as RNA Viruses (which are considered not to be living organisms) and forms of non DNA based inheritance.

DNA doesn't operate in isolation, for example the same DNA sequence in a human produces a multitude of specialised cells. It instead acts in combination with both the environment it exists within and the environments it has existed within. Hence it is more correct to say that DNA contains genetic information that influences the phenotype (characteristics) of an organism.

To keep things simple, I'll ignore the multitude of RNA types (from messenger to transport), the issues of expression, the terminology of genes and 3D geometry and take a few chunky liberties in the description of how DNA works.

In principle, DNA consists of a long double helix sequence of four basic nucleotides (the base pairs) known as C,G,A,T. Different sections of this sequence (referred to as genes) are transcribed and translated into protein structures which affect the operation of the cell. Each three letter word (a codon) of the genetic sequence (i.e. CGT or GAT, giving 64 possible combinations) is translated to an amino acid (of which there are 22 standard).

The entire complexity of life is built upon such simple subsystems which in turn are part of ever more complex systems. Without this component structure, the level of complexity in living organisms would not have been feasible. It's worth noting that the agility of complex structures to evolve is dependent upon the organisation of their subsystems.

So, what has this to do with cloud?

Well, if you take an example such as Amazon's Web Services, the complexity of the many systems that users have developed with cloud services is based upon the provision of simple, standard subsystems for storage, compute resources and networks.

There is some limited variability in the type of subsystems (for example the size of Amazon instances) and the introduction of a new Cluster Compute Instance but these are the genetic analogy to amino acids which are then used to build more complex protein structures. Your deployment scripts (whether you use a system such as RightScale or another) are your DNA which is then transcribed and translated into the deployment of basic instances to create the complex structures you require.

So, back to James' post. My objection to the post is that whilst you, as a user, can create a slime mould or a neuron or a million other cellular anologies with these basic components, the key is how YOU combine these common and well defined (i.e. commodity-like) components.

James' however infers in his post that we need to see alternative cloud models, not just the "slime mold model cloud" but "more complex topologies" with the "emergence of more topologically calibrated and therefore feature rich clouds". In principle he is arguing for more configuration of these basic subsystems.

Whilst I agree that some additional basic subsystems (e.g. the cluster computer instance) are needed, I'd argue against the principle of wide ranging configuration in the subsystems.

Whilst such a richness of diversity in the underlying subsystems does create benefits for technology vendors - which company hasn't fallen for the "competitive advantage of customizing a CRM system" gag - it will not create the "wealth" and diversity in user created systems. Instead, it's more likely to create further lock-in issues, move us away from competitive marketplaces and end up with users spending vastly more time building and configuring stuff which really doesn't matter.

There are always edge cases, however in general the range of subsystems we require are fairly limited and it's from these we can build all the different type of systems (or cells) we want.

If there is anything that should be learned from biological analogies, it is from such "modest entities", such simple subsystems that complexity and diversity is created. We've learned this lesson throughout time, from the electronic to the industrial revolution to the works of Vitruvius.

Unfortunately, as the old adage goes - "the one thing you learn from history, is we never learn from history".

One final note: analogies between cloud computing and biological systems are generally weak at best - my above example is no exception. I use it purely to continue in the same spirit as the original discussion and to try and highlight the core issue of diversity in the subsystems vs diversity in what is built with stable subsystems. I don't recommend comparing cloud computing to biology, unless you want endless arguments.

One very final note: computer models are based on simple arithmetic and hence are bound to Goedel's law of incompleteness, neither being being both complete and certain. As activities provided through software tend towards being ubiquitous and well defined, they will tend towards being a good enough component like a defined brick with standardised interfaces. The components themselves will have some inherent non-linear qualities (i.e. the halting problem) which is why the design for failure paradigm is so important.

Biological components are also only linear at a superficial level and on an individual level they are highly non-linear and cannot be described by simple arithmetic means nor modeled with certainty by a computer. I suspect, this is why virtual worlds tend to only seem "real" when populated by real people. The interface might be standardised and the avatars can be seen as having a set limit of linear capabilities but the actual component (i.e. the person) is highly non-linear.

For these reasons, biological system have evolved highly complex regulatory systems (such as the human immune system) which even today we barely understand. However, we're acutely aware that a major function of it is to control rogue cells. This level of complexity is far beyond the capabilities of modern computing and also filled with numerous information barriers (the uncertainty principle is only one of many) which prevent us from anything more than approximation. For these reasons, I find some useful concepts in biological systems (Red Queen, Ecosystems, Co-evolution etc) along with certain derived concepts, such as design for failure, but direct comparison is filled with danger.

A life less cloudy

2010-06-02T02:32:00.004+01:00

For 18 months I ran the cloud strategy at Canonical. It was a real pleasure to work with some very amazing people and play a small part in Canonical's journey. Today, Ubuntu has become the dominant cloud operating system and leads the field in provision of hybrid clouds combining both public use on Amazon EC2 with private cloud technology.

However, I've been in the cloud space for five years, I've got my long service medal and it's time to scratch another itch but I thought I'd wrap up with some general thoughts.

Definition of Cloud

Given that 200 years after the industrial revolution we still can't agree on a good enough definition of that, it was never likely the industry was going to agree on cloud. The problem with cloud is it's not a thing but a transition of certain IT activities from a product to a more service based economy provided through large scale utilities. This change was predicted by Douglas Parkhill in his 1966 in his book the "Challenge of compute utilities". He made the comparison to the electricity industry and talked of a future of public, private, community and government utilities providing computer resource online, on demand, with elastic supply and charged on a utility basis. These characteristics permeate cloud today but just don't try and get everyone to agree.

Why now?

A number of factors were needed before this change could occur. Firstly, you needed the concept which Parkhill kindly provided us with. Secondly, you needed the technology to achieve this service provision but we've had that for the best part of a decade. Thirdly, you needed those IT activities to be well defined and ubiquitous enough to be suitable for the volume operations needed to support a utility business. This has also happened over the last decade.

Lastly, you needed a change in attitude and a willingness of business to adopt these new models of provision. This change in business attitude started with Strassman in the 1990s when he pointed out that spending on IT amounted to little more than an arms race with dubious links to any value created. Nick Carr developed these concepts further and this wisdom has now peculated throughout the business world.

Is everything suitable for the cloud?

The answer to this is "depends". To understand why we must first appreciate that IT consists of many ubiquitous and well defined activities that are suitable for cloud provision. However, IT also contains many new activities that are not ubiquitous nor well defined and hence aren't suitable for provision as cloud services. This of course doesn't mean that such novel activities can't be built on cloud services.

There is a difference between that which can be provided as a cloud service and that which can be built upon cloud services. However, don't get comfortable yet because activities have a lifecycle. Business activities themselves undergo a constant transition from innovation to commodity. In short that which can be built upon cloud services will eventually become that which is provided as cloud services.

Benefits & Risks.

The benefits and risks of cloud have been listed countless times before so I'll avoid going through these again and just point out some highlights.

First, cloud is fundamentally about enabling and increasing rates of user innovation (a combination of creative destruction and componentisation). Don't get bogged down into the cost saving arguments because whilst cloud will increase efficiency, a number of factors will explode consumption.

Second, you don't have choice over cloud. Your company is competing in an ecosystem with others who will adopt cloud computing. The benefits of cloud will create pressure for you to adopt in order to remain competitive (this is the "Red Queen Hypothesis")

Third, you do have a choice over how to implement cloud. The hybrid model of public & private service is a standard supply chain trick of balancing benefits vs risks. Expect lots of this and ignore anyone who tells you that private cloud isn't cloud computing.

Fourth, it's not about virtual data centres. This one is a tough nut to crack since lots of people have spent vast sums on virtual data centres and they want them to be infrastructure clouds. The problem is simply one of economics. VDCs are based upon concepts of resilience which have their origin in physical product world and come with a wide range of overheads. Infrastructure clouds are based upon the economics of volume operations with vast numbers of cheap & reliable enough VMs and resilience created through design for failure at the management layer.

Fifth, providers ain't your buddies. In a commodity world of utility services then the market, the exchanges, brokers, standardisation and assurance bodies (providing price vs QoS comparisons) are your buddies. Whilst some providers get this, others will try and convince you of their innovative cloud features. These are the enemies of portability & easy switching and the fastest way to end up paying more than you need.

Six, security, auditing and governance have to change in the cloud world and it's not going to be the same as the perimiterised, defense in depth approach of old.

There's a whole host of topics to talk about from the future formation of exchanges, the creation of rating agencies, the introduction of insurance and derivative instruments, the componentisation effects of the platform space, the growth in importance of orchestration, the …. but this has been done to death for the last five years.

I'll mention one last thing. The secret to Ubuntu's success in the cloud is execution.

Rather than fighting battles which couldn't be won, Canonical choose to adopt the dominant approach in the market, provide users with choice and real technology and then combine this with a clear story. Hence Canonical launched a hybrid cloud strategy making it easy for users to build with Ubuntu in the public cloud computing space combined with simple and easily installed technology for creating private clouds that matched the same APIs.

By combining these approaches, management tools could be used across both public and private clouds with machine images that could run in both environments. Users had their first real opportunity to exploit a hybrid cloud environment, choosing for themselves the right balance of public and private with an open source technology that limited lock-in and took advantage of both the Ubuntu and the Amazon EC2 ecosystems. Since it's launch in April '09. this approach has gone from strength to strength and has created a beach-head for Canonical to extend into the orchestration, platform and marketplace fields.

However, the strategy wasn't new, what was critical was the way in which Canonical's excellent engineers achieved this.

Despite the various prognostications on the disruptive nature of cloud, it's old hat. The strategies around cloud were formulated many many years ago. The concepts and ideas around cloud, orchestration, platforms, the economics, the barriers to adoption and how to play in this game are all well known.

If you're looking for that great strategy to bring your company into the cloud, be aware that whatever you think of is almost certainly being done by others and has been contemplated to the nth degree by many more. Today, success in cloud is all about partnerships, execution, tactical moves and timing in a great game of shogi that is already well advanced.

The time for strategists in the cloud space is over and it's all about how the game is played.

Don't get me wrong, the game is still an exciting place. There are all manner of tactical plays, twist and turns and efforts to help standardise and form this future industry. However, strategy is my passion, it's where my heart lives.

Which is why I have joined CSC's Leading Edge Forum to pursue a new challenge at the very centre of change.

VMForce, Zimki and the cloud.

2010-05-01T01:39:00.012+01:00

Before discussing VMForce, I first want to remind people of the tale of Zimki.

Zimki was one of the first platform as a service offerings, initially built and released in 2005 under the name libapi (liberation API).

With Zimki, a user could sign onto this hosted service and create entire applications (including all client and server side components) in one language - JavaScript. You developed inside the platform consuming the services it provides and the user had no need to be aware of physical machines. The service was charged on a utility basis including operations, storage and network.

The use of JavaScript as the base language was made for several reasons. First, it was widely known. Second, it had been contained within the browser for almost a decade. Thirdly, with one common language there was a reduced potential for translation errors between client and server side code.

The platform provided additional primitives to the language for storage of objects (data), templating systems and simple conversion of functions to web services. Building a system in Zimki was simple with entire applications being written and released in a matter of hours. However, this wasn't some sort of magic but instead a normal consequence of componentisation and a standardised platform delivered as a service.

The entire platform had an extensive API set for its management, monitoring and development which was used both by the web based development tools and the local development environment. Since all code and data were stored as objects, moving entire applications from one hosted Zimki environment to another hosted Zimki environment was relatively trivial and was demonstrated several times in '06/'07.

The entire system had also been constructed to allow for the rapid development of shared code bases & components amongst its users. Zimki was far from perfect but it was highly useable and was rapidly growing.

During 2006, it was announced that Zimki would be entirely open sourced in 2007 to enable other companies to run their own Zimki installations and for other providers to set-up. The purpose of this action was to create a competitive marketplace of Zimki providers (multiple public sources) as well as enabling hybrid clouds (the use of public and private provision).

The system was going to be GPLv3'd precisely because of the "SaaS" loophole. The intention was to allow multiple providers to make operational improvements to the code base for reasons of service competition whilst ensuring portability through a separately established assurance authority with a trademarked compliance stamp. As part of this, an exchange was to be established.

The use of open source, extensive APIs (covering management, development and monitoring), all objects and code freely portable and an assurance authority was specifically designed to create a competitive marketplace.

Everything was looking good. Zimki was backed by a well resourced company, Fotango, for which I was the acting CEO. We'd recorded 16 highly profitable quarters, had significant cash reserves and were ready to take on this market.

Unfortunately, the parent company Canon had reasonably decided that this utility computing world was not core to its objectives and it had a different focus. The consequence of this was that Zimki was never open sourced, the service was terminated and the company outsourced.

Had Canon taken another path then it could well have become one of the largest cloud providers in today's market, rivaling those of later entrants such as Google (with AppEngine), Microsoft (with Azure) and VMForce. But that's innovation for you, it's a gamble and outcomes are uncertain.

Bizarrely, the termination of Zimki reinforced the importance of portability and a choice of providers in the PaaS space.

Probably the most important lesson learned from Zimki was that lock-in can be created in multiple forms, including lack of access to code & data, high exit costs, additional services and management tools. The only viable way of preventing this and creating a marketplace with competition in service provision, is if the entire service is open sourced and built with the ideas of portability from the beginning.

So five years later, we have the announcement of VMForce providing a PaaS offering based around Java. Obviously it will provide a much faster rate of development and deployment (a normal consequence of componentisation) along with all the other benefits of a cloud service. This however, isn't anything new or pioneering, it's a consequence of a standardised platform which we showed back in 2006.

Unfortunately the system won't be open sourced, it runs on a proprietary platform and there exists many additional services around it. These are all ample opportunities for lock-in.

So we get the standard benefits and we get lock-in, when we could just get the benefits. Unfortunately this won't happen and we won't see freely competitive marketplaces until the underlying reference models (i.e. the entire system) are open sourced and people realise that open standards are necessary for portability but not sufficient for it.

We knew this in 2005, nothing has altered my opinion since then.

Instead, I expect we will hear lots of open talk without the essential ingredient of actually being open sourced.

So my thoughts on the VMForce announcement, nothing new and the same old problems. I'm unimpressed so far but at least it gives us another Java platform to play with.

For reference, this is an old Zimki Presentation from 2006, with the talk notes included as text. There are a couple of flaws in the concepts which were later refined, but the basics are all there.

Zimki 2006

View more presentations from swardley.

Use cloud and get rid of your sysadmin.

2010-04-12T14:46:00.013+01:00

Following on from my Cloud Computing Myths post.

The principle argument behind cloud getting rid of sysadmins is one of "pre-cloud a sysadmin can manage a few hundred machines, in the cloud era with automation a sysadmin can manage tens of thousands of virtual machines". In short, since system admins will be able to manage a two orders of magnitude greater number of virtual machines then we will need less of them.

Let's be first clear what automation means. At the infrastructure layer of the computing stack there are a range of systems, commonly known as orchestration tools, which allow for basic management of a cloud, automatic deployment of virtual infrastructure, configuration management, self-healing, monitoring, auto-scaling and so forth. These tools take advantage of the fact that in the cloud era, infrastructure is code and is created, modified and destroyed through APIs.

Rather than attempting to create specialised infrastructure, the cloud world takes advantage of a bountiful supply of virtual machines provided as standardised components. Hence scaling is achieved not through provision of an ever more powerful machine but deployment of vastly more standardised virtual machines.

Furthermore the concept of a machine also changes. We're moving away from the idea of a virtual machine image for this or that, to one of a basic machine image and all the run time information you require to configure it. The same base image will become a wiki, a web server or part of a n-tier system.

All of these capabilities allow for more ephemeral infrastructure, rapidly changing according to need with rapid deployment and destruction. This creates a range of management problems and hence we have the growth of interest in orchestration tools.

These tools vary from specifically focused components to more general solutions and include chef, controltier,CohesiveFT, capistrano, rightscale, scalr and the list goes on.

A favourite example of mine, simply because it acts as a pointer towards the future, is PoolParty. Using a simple syntax of describing infrastructure deployment, PoolParty synthesises the core concepts of this infrastructure change. For example, deploying a system no longer becomes a long architectural review and planning process, an RT ticket requesting some new servers with an inevitable wait, the installation, racking and configuration of those servers followed with change control meetings.

Deploying a system becomes in principle as simple as :-

Pool "my_application" do
Cloud "my_application_server" do
Using EC2
Instance 1...1
Image_id "xxxxx"
Autoscale
end

Cloud "my_database_server" do
Using EC2
Instances 1...1
Image_id "xxxxx"
end
end

It is these concepts of infrastructure as code and automation through orchestration tools when combined with a future of computing resources provided as larger components (pre-built racks and containers) which have led many to assume that cloud will remove the roles of many sysadmins.

This is a weak assumption.

A historical review of computing resource usage shows a price elasticity of between |2| to |3|. In short, as the cost for provision of a unit of compute resource has halved the demand has increased between 100-150% leading to today's proliferation of computing.

Now, depending upon who you talk to, the inefficiency of computer resources in your average data centre runs at 80-90%. Adoption of private clouds should (ignoring the benefits of using commodity hardware) provide a 5 x reduction in price per unit. Based upon historical precedents, you could expect this to lead to a 10-15x increase in consumption as we find the long tail of applications that companies desire becomes ever more feasible.

Of course, this ignores transient applications (those with a short life time such as weeks, days or hours), componentisation (e.g. self service and use of infrastructure as a base component), co-evolution effects (i.e. the creation of higher order systems) and the larger economies of scale potentially available on public providers.

Given Moore's law, the current level of wastage, a standard VM / Physical server conversion rate, greater efficiencies in public provision, increasing use of commodity hardware and the assumption that expenditure of computing resources will remain flat (any reductions in cost per unit being compensated by increase in workload) then it is entirely feasible that within 5-7 years these effects could lead to a 100x increase in virtual infrastructure (i.e. number of virtual servers compared to current physical servers).

It's more than possible that in five years time every marketing department will have its own 1,000 node hadoop cluster for data processing.

So, we come back to the original argument which is "pre-cloud a sysadmin can manage a few hundred machines, in the cloud era with automation a sysadmin can manage tens of thousands of virtual machines". The problem with this argument is that if cloud develops as expected then each company will be managing two orders of magnitude more virtual machines which means there'll be at least as many sysadmins as there are today.

Now whilst the model changes when it comes to platform and software as a service (and there are complications here which I'll leave to another day), the assumption that cloud will lead to less system adminstrators is another one of those cloud myths which hasn't been properly thought through.

P.S. The nature of the role of a sysadmin will change and their skillsets will broaden, however if you're planning to use cloud to reduce their numbers then you might be in for a nasty shock.

P.P.S. Just to clarify, I've been asked by a company which runs 2,000 physical servers whether this means that in 5-7 years they will be running 200,000 virtual servers (some of which will be provided by private and others on public clouds, ideally through an exchange or brokers). This is exactly what I mean. You're going to need orchestration tools just to cope and you'll need sysadmins to be skilled in these and managing a much more complex environment.

Common Cloud Myths

2010-04-09T02:29:00.002+01:00

Over the last three years, I've spent an increasingly disproportionate amount of my time dealing with cloud myths. I thought I'd catalogue my favourites by bashing one every other day.

Cloud is Green

The use of cloud infrastructure certainly allows for more efficient provision of infrastructure through matching supply to demand. In general :-

1. For a traditional scenario where every application has its own physical infrastructure then each application requires a capacity of compute resources, storage and network which must exceed its maximum load and provide suitable spare capacity for anticipated growth. This situation is often complicated by two factors. First, most applications contains multiple components and some of those often highly under utilise physical resources (for example load balancing). Second, due to the logistics of provisioning physical equipment then the excess capacity must be sufficiently large. At best, the total compute resources required will significantly exceed the sum of all the individual peak application loads and spare capacity.

2. The shared infrastructure scenario covers networks, storage and compute resources (through virtualisation). Resource requirements are balanced across multiple applications with variable loads and the total spare capacity held is significantly reduced. In an optimal case the total capacity can be reduced to a general spare capacity plus the peak of the sum of the application loads. Virtual Data Centres, provisioning resources according to need, are an example of shared infrastructure.

3. In the case of a private cloud (i.e. a private compute utility), the economics are close to that of a shared scenario. However, there is one important distinction in that a compute utility is about commodity infrastructure. For example, virtual data centres provide highly resilient virtual infrastructure which incur significant costs whereas a private cloud focuses on rapid provision of low cost, good enough virtual infrastructure.

At the nodes (the servers providing virtual machines) of a private cloud, redundant power supplies are seen as an unnecessary cost rather than a benefit. This ruthless focus on commodity infrastructure provides a lower price point per virtual machine but that necessitates that resilience is created in the management layer and application (the design for failure concept). The reasoning for this, is the same reasoning behind RAID (redundant array of inexpensive disks). By pushing resilience into the management layer and combining more lower cost, less resilient hardware you can actually enable higher levels of resilience and performance for a given price point.

However, the downside is that you can't just take what has existed on physical servers and plonk it on a cloud and expect it to work like a highly resilient physical server. You can however do this with a virtual data centre.

This distinction and focus on commodity provision is the difference between a virtual data centre and a private cloud. It's a very subtle but massively important distinction because whilst a virtual data centre has the benefit of reducing educational costs of transition in the short term (being like existing physical environments), it's exactly these characteristics that will make it inefficient compared to private clouds in the longer term.

4. In the case of a public cloud infrastructure (a public compute utility), the concepts are taken further by balancing variable demands of one company for compute resources against another. This is one of many potential economies of scale that can lead to lower unit costs. However unit cost is only one consideration here, there are transitional and outsourcing risks that need to be factored in which is why we often use hybrid solutions combining both public and private clouds.

The overall effect of moving through these different stages is that the provision of infrastructure becomes more efficient and hence we have the "cloud is green" assumption.

I pointed out, back in 2008 at IT@Cork, that this assumption ignored co-evolution, componentisation and price elasticity effects.

By increasing efficiency and reducing cost for provision of infrastructure, a large number of activities which might have once not been economically feasible become economically feasible. Furthermore, the self-service nature of cloud not only increases agility by enabling faster provision of infrastructure but accelerates user innovation through provision of standardised components (i.e. the infrastructure equivalent of a brick). This latter effect can encourage the co-evolution of new industries in the same manner that the commoditisation of electronic switching (from the innovation of the Flemming valve to complex products containing thousands of switches) led to digital calculators and computers which in turn drove further commoditisation and demand for electronic switching.

The effect of these forces is that whilst infrastructure provision may become more efficient, the overall demand for infrastructure will outstrip these gains precisely because infrastructure has become a more efficient and standardised component.

We end up using vastly more of a more efficient resource. Lo and behold, cloud turns out not to be green.

The same effect was noted by Willam Stanley Jevons in the 1850s, when he "observed that England's consumption of coal soared after James Watt introduced his coal-fired steam engine, which greatly improved the efficiency of Thomas Newcomen's earlier design"

Cloud computing made simple

2010-03-25T11:04:00.019Z

It has been a truly amazing year since we embarked on our "cloud" journey at Ubuntu, hence I thought I'd review some of the highlights.

We started the journey back in 2008 when Mark Shuttleworth announced our commitment to providing cloud technology to our users. At that time, the cloud world was already in a state of growing confusion, so we adopted an approach of :-

make the cloud simple.
focus on one layer of the computing stack (infrastructure) to begin with.
give our users real technology not promises.
help drive standardisation (a key requirements of this shift towards a service world) by adopted public defacto standards.
work with leading partners in this growing industry.
provide open source systems to avoid lock-in issues.
actively work to mitigate risks and concerns over cloud by giving our users options.

Hence, in April'09 as part of Ubuntu 9.04 we launched our hybrid cloud strategy.

Our approach was based around the adoption of Amazon EC2 / S3 & EBS as the public defacto standard rather than the creation of some new APIs (there's too many already).

We provided Ubuntu images for use on Amazon EC2 (public cloud) and the technology to build your own private cloud (known as Ubuntu Enterprise Cloud) that matched the same APIs of Amazon. We also added management tools which could cross both public and private domains because of our adoption of a standard API set.

For 9.10 we significantly improved the robustness and ease of setting up a private cloud (Mark built his own several node system in under 25 mins from bare metal). We provided the base for an application store, improved the management capabilities of Landscape and the features of UEC grew extensively. We also launched training, consultancy and support services for the cloud and a JumpStart program to help companies move into the cloud quickly.

During this time we've worked closely with many partners, I'll mention a few (more details can be found on the Ubuntu Cloud site) :-

Eucalyptus whose open source technology we adopted into the distribution as a core part of Ubuntu Enterprise Cloud.
Intel's Cloud Builder program to provide best practices on how to create a private cloud using UEC. I'd strongly recommend reading the whitepaper.
RightScale & CohesiveFT to provide best of breed public management tools alongside our own Landscape system.
Dell, who will offer a range of pre-built clouds using a series of ‘blueprint’ configurations that have been optimised for different use cases and scale. These will include PowerEdge-C hardware, UEC software and full technical support.

In one year, we've made cloud simple for our users. We've brought our "Linux for Humans" philosophy into the cloud by getting rid of complexity, confusion and myth.

If you want to get into cloud, then we offer :-

Simple Choices: You can have either private, public or hybrid (i.e. public + private) infrastructure clouds.
Simple Setup: If you want to build a private cloud, then Ubuntu makes the set-up ridiculously easy. You can be up and running with your own cloud in minutes. Along with with our community documentation covering CD installation and more advanced options, you can also find detailed information on how to build a cloud through Intel's cloud builder program. However, if building a cloud still sounds too daunting then Dell offers pre-built, fully configured and supported private clouds.
Simple Management: You can use the same tools for both your private and public clouds because we've standardised around a common set of APIs. There's no need to learn one set of systems for private and another for public.
Simple Bursting: Since we provide common machine images which run on both public and private cloud offerings combined with standardised APIs, then the process of moving infrastructure and combining both private and public clouds is ... simpler.
Enterprise Help: If you still need help then we offer it, including 24x7 support and a jumpstart program to get your company into the cloud.
Open source: UEC, the Ubuntu machine images and all the basic tools are open sourced. We're committed to providing open source systems and following through on a genuine open source approach i.e. the system is open source and free and so are all the security patches and version upgrades.

The results of this year have been very encouraging. We recently estimated that there are now over 7,000 private clouds built with UEC, however with 7% of users in our annual Ubuntu User Survey saying that they have built a UEC cloud, the true figure might be very much higher. It was great to hear that almost 70% of users felt Ubuntu was a viable platform for the cloud but there were several surprising statistics including :-

54% were using the cloud in some form or another (software, platform or infrastructure). However giving the fuzziness of the term cloud, this can only be seen as a signal of intent to use online services.
Only 10% had used public cloud providers (such as Amazon) for infrastructure. What was quite remarkable was that given the relatively recent availability of UEC, almost as many people had built private clouds as had used public cloud providers.
60% felt that the use of private cloud was more important to their organisation, 25% thought that both private and public was of equal importance whilst only 15% felt that public cloud was the most important.

Whilst this survey was targetted at Ubuntu users, the data we receive from external sources suggest that Ubuntu is becoming the dominant operating system for consumers of the infrastructure cloud space. Even a simple ranking of search terms using Google's Insight around cloud computing show how significant a player Ubuntu is.

Whilst this is great news, what really pleases me is that we're making cloud simple and real for organisations and listening to what they need. We're getting away from the confusion over cloud, the tireless consultant drivel over whether private cloud is cloud computing and the endless pontifications and forums debating vague futures. Instead, we're giving real people, real technology which does exactly what they want.

Over the next year we're going to be tackling issues around creating competitive marketplaces (i.e. more choice for our users), simplfying self-service IT capabilities and orchestration and providing a wide range of open source stacks and platforms to use in the cloud.

We're going to continue to drive down this path of commoditisation by providing common workloads for the cloud (the same as we've been doing for server) and helping businesses to standardise that which is just cost of doing business.

Regardless of any attempts to badge "cloud" as just a more advanced flavour of virtualisation or describe it as "not real yet" by various late vendors, we will be doing our best to bust the various "cloud" myths and push the industry towards competitive marketplaces of computer utilities through defacto standardisation.

Commoditise! Commoditise! Commoditise!

I'm also delighted about our partners successes, with RightScale passing the million server mark, Amazon's continual growth and leadership with the introduction of spot markets, Dell's outstanding move to make cloud mainstream, Intel's push to make cloud easier & Eucalyptus' continued adoption and the appointment of Marten Mickos as CEO.

P.S. if you want to keep track of what's happening with Ubuntu in the cloud, a good place to start is our cloud blog or following our twitter feed, ubuntucloud.

Cloud rant

2010-03-14T16:47:00.012Z

For the last two posts, I've had a pop at the term "cloud", I'd like to now explain my reasoning in more detail.

First, as always, some background information.

What we know

The transition from a product to a services world in I.T. is very real. It's happening now because of the confluence of concept, suitability, technology and a change it business attitude. Overall it's driven by the process of commoditisation and it is the very ubiquity of specific I.T. activities that makes them potentially suitable for volume operations. I say potentially because volume operations is only viable for activities which are both well defined and ubiquitous.

Fortunately ubiquity has a relationship to certainty (or in other words how well understood, defined and therefore certain an activity is) which is why these activities are suitable for provision on the basis of volume operations through large computer utilities.

Lifecycle

For many years I've talked about lifecycle and the evolution of business activities. Any activity goes through various stages from its first innovation (as per the use of computer resources in the Z3 in 1941) to custom built examples to products which describe the activity. Usually, the activity ends up becoming a ubiquitous and well defined commodity (assuming there are no natural limits or constraints).

During this lifecycle, as the activity becomes more defined in the product stage, service models can often arise (for example the rental model of the managed hosting industry for computing infrastructure or the early subscription like models of electricity provision). As the activity becomes more of a commodity the existence of these service models tends to lead to the rise of utility services (as with electricity provision).

An essential requirement for the growth of the utility model (and the type of volume operations necessary to support it) is that consumers view that what's provided is a commodity. It's little more than a cost of doing business and a standardised version is good enough.

The latter is why a change of attitude is critical in development of the utility service model. If, for example, consumers still view the activity as creating some form of competitive advantage (whether true or not), they are unlikely to adopt a utility model of standard services.

The change in attitude

Over the last decade the attitude of business towards certain I.T. activities has changed dramatically. Recently, a group of 60 odd CIOs & Architects highlighted that many of the I.T. related activities they undertook were commonplace and well defined, particularly across their industry & geography.

Now that doesn't mean they did things the same way, quite the reverse.

Taking just one activity, ERP, then all of these companies agreed that whilst they gain no competitive advantage in ERP, it was an essential cost of doing business. They also agreed that they all had their own customised processes for ERP which they invested heavily in. The shock was their agreement that these different processes provided no differential benefit. It was estimated that for this one activity alone, across this small group of companies, then $600 million p.a. was spent maintaining differences which provided no value.

By reducing customisation through the provision of ERP as standardised services, then each company would significantly benefit in terms of cost savings. Standardisation of processes and removing costs associated with customisation is seen as one of the major benefits of the transition from an "as a Product" to an "as a Service" world.

Let's be clear here, "as a Service" was considered shorthand for "provision of a commodity activity through standardised services via a competitive marketplace of computer utilities". These companies were not looking for an outsourcing arrangement for a highly customised service for their needs, they were looking for a commodity to be treated as a commodity.

The concepts of computer utilities offering elastic and infinite supply on demand, provision of activities through services and commoditisation are all tightly coupled.

The benefits & risks of "cloud"

The benefits of this shift towards service provision via large computer utilities has been discussed extensively for the last 40 years :-

economies of scale (volume operations)
focus on core activities (outsourcing to a service provider)
pay per use (utility charging)
increased consumer innovation ( componentisation)

However, one critical benefit that gets missed is standardisation itself.

The risks associated with this change (ignoring the disruptive effect on the product industry) can be classified into transitional risks (related to the change in business relationship) and generic outsourcing risks. I've categorised these below for completeness.

Transitional Risks

Confusion over the new models.
Trust in the new providers.
Transparency of relationships.
Governance of these new models (including auditing, security & management).
Security of supply

Outsourcing Risks

Suitability of the activity for service provision.
Vendor lock-in (& exit costs).
The availability of second sourcing options.
Pricing competition.
Loss of strategic control.

Transitional risks can be mitigated through standard supply chain management techniques. For example with electricity we often combine both public and private sources of provision (a hybrid option). However outsourcing risks require the formation of competitive marketplaces in order to mitigate. Whilst the latter is a genuine concern for companies, even without these marketplaces the benefits of this change are still attractive.

The problem with the term "Cloud"

This change in I.T. is all about standardised service provision of commodity activities through a competitive marketplace of computer utilities. The notion of utility conjures up easily understood and familiar models.

Few would have a problem in understanding how access to a standardised form of electricity through a marketplace has allowed for a huge range of new innovations built upon consuming electricity. Few would have a problem in understanding how the providers themselves have sought new innovative ways of generating electricity. Few would ever consider electricity itself as a form of innovation, to most it comes from a plug and it is critical that it is standardised.

The problem with the term "cloud" beyond being fuzzy, it is often used to describe this change in I.T. as something new and innovative. The term helps disguise a fundamental shift towards a world where the bits don't matter and it's all about services. The term allows for all manner of things to be called cloud, many of which have little to do with the standardisation of an activity and its provision through utility services.

You could easily argue the term is misleading as it encourages customisation and distracts the consumer from what should be their focus - standardised services through a competitive marketplace of computer utilities.

Alas, as I said we ALL have to use the term today because of its momentum.

At Ubuntu we focus on commodity provision of activities (common workloads), providing our users with the technology to build a private computer utility (nee "Cloud", as part of a hybrid strategy), the adoption of the defacto standard of EC2 & S3 and we also provide all the technology as open source to encourage the formation of competitive markets.

We use the term "cloud" because it's what customers, analysts and others expect to hear. This of course doesn't stop us from explaining what is really happening and busting the "cloud" myths that exist.

Is your cloud a poodle?

2010-03-13T12:14:00.005Z

Since we're fond of replacing meaningful concepts such as commoditisation, lifecycle, categorisation and computer utilities with bland terms like "cloud", I thought I'd follow the trend on to its next logical conclusion - Poodle Computing.

The shift of I.T. activities from being provided "as a Product" to being provided "as a Service" through large computer utilities has an obvious next step - the formation of competitive marketplaces. These marketplaces will require standardisation of what is after all a commodity (i.e. ubiquitous and well defined enough to be suitable for service provision through volume operations) and the ability of consumers to switch easily between and consume resources over multiple providers (which in turn requires multiple providers, access to code & data, interoperability of providers and an overall low exit costs)

I won't bore you with the mechanics of this and the eventual formation of brokerages & exchanges, I covered this subject extensively in 2007 when I made my "6 years from now you'll be seeing job adverts for computer resource brokers" prediction.

However, in this future world of brokerages and fungible compute resources (or fungitility as I jokingly called it) the consumer will become ever more distanced from the source of provision. This will be no different to the many other forms of utilities where vibrant exchange markets exist and what the consumer purchases often has gone through the hands of brokers. You don't actually know which power station generated the electricity you consume.

So this brings me to the title of the post. As consumer and the source becomes more distanced, it reminds me of Peter Steiner's cartoon"On the Internet, nobody knows you're a dog".

On that basis, what sort of dog flavour of computing resource will you be consuming?

By introducing the concept of "dog computing" to cover this "cloud of clouds" world (hey, they're both meaningless) then the marketing possibilities will become endless and a lot more fun.

I can see the conversation now, walking into a lean and mean sales organisation and saying to the CEO that they are using "Poodle Computing". Shouldn't they be using our brand new "Pitbull Computing" or at least upgrading to "Springer Spaniel"?

We could always call things what they are (computer utilities & competitive markets of computer utilities") but I suspect we will end up with "cloud of clouds", "cloud exchanges" and an OTC market of ominous sounding "cloudy futures".

What is Cloud?

2010-03-12T13:35:00.022Z

Before we can discuss this term, a bit of history and background is needed.

Activities

All business activities undergo a lifecycle, they evolve through distinct stages including :-

the first introduction of a new activity (its innovation)
the custom built examples replicating this activity (the copying phase)
the introduction of products which provide that activity ( the product stage, including numerous rounds of feature differentiation which are also unfortunately called product innovation)
The activity becoming more of a commodity (ubiquitous, well-defined and with no qualitative differentiation). In certain circumstances that commodity can be provided through utility services.

It should be noted that the characteristics of an activity changes as it move through its life-cycle. As a commodity it's of little strategic value (or differentiation) between competitors whereas in its early stages it can often be a source of competitive advantage (a differential).

Information Technology

At any one moment in time, I.T. consists of a mass of different activities at different stages of their life-cycle. Some of those activities are provided through discrete software applications (an example might be ERP), other activities relate to the use of platforms (developing a new system using RoR or provisioning of a large database etc) whilst others relate to the provision of infrastructure (compute resource, storage, networks).

You can categorise these activities into a computing stack of infrastructure, platform and software. Of course you can go higher up the stack to describe the processes themselves and beyond, however for this discussion we will just keep it simple.

What's happening in IT today?

Many activities in I.T. that were once innovations but more recently have been provided as products (with extensive feature differentiation) have now become so ubiquitous and so well defined that they have become little more than a commodity that is suitable for service provision. You can literally consider that chunks of the "computing stack" are moving from an "as a Product" to an "as a Service" world.

This change is the reason why we have the "Infrastructure as a Service" to "Platform as a Service" to whatever else "as a Service" industries. Of course, there are many higher order layers to the stack (e.g processes) but any confusion around the "as a Service" term generally only occurs because we never used to describe these activities with the "as a Product" term.

Had we categorised the previous software industry in terms of "Software as a Product", "Platform as a Product" etc, then the change would have been more obvious.

Why now?

This change requires more than just activities being suitable for utility service provision. It also requires the concept of service provision, the technology to achieve this and a change in business attitude i.e. a willingness of business to adopt these new models. Whilst the concept is old (more on this later), and the technology has been around for some time (yes, it has matured in the last decade but that's about all), both the suitability and change of business attitude are relatively new.

Thanks to the work of Paul Strassman (in the 90's) and then Nick Carr (in the 00's), many business leaders have recognised that not all I.T. is a source of advantage. Instead much of I.T. is a cost of doing business which is ubiquitous and fairly well defined throughout an industry.

It was quite refreshing to recently hear a large group of CIOs, who all spent vast amounts of money maintaining highly customised CRM systems, comment that actually they were all doing the same thing. These systems provided no strategic value, no differential and in reality what they wanted was standardised, low cost services charged on actual consumption basis for what is essentially a cost of doing business. They also wanted this to be provided through a marketplace of service providers with easy switching between them.

This is quite a sea change from a decade ago.

The change from a "as a Product" to an "as a Service" world is happening today because we have the concept, technology, suitability and most importantly this changing business attitude.

An old Concept

The concept of utility service provision for I.T. is not new but dates back to the 1960's. Douglas Parkhill, in this 1966 book - "The Challenge of the Computer Utility" - described a future where many computing activities would be provided through computer utilities analogous to the electricity industry. These computer utilities would have certain characteristics, they would :-

provide computing resources remotely and online
charge for the use of the resources on the basis of consumption i.e. a utility basis
provide elastic & "infinite" supply of resources
benefit from economies of scale
be multi-tenanted

Douglas noted that these computer utilities would take several forms as per the existing consumption of other utilities. These forms included (but are not limited to) public, private & government utilities. He also noted that eventually we would see competitive markets of computer utilities where consumers could switch providers, consume resources across multiple providers (i.e. a federated use) and consume all manner of hybrid forms (e.g. private and public combinations)

One final note is the term utility means a metered service where the charge is based upon consumption. That charge might be financial or it could be in any other currency (e.g. access to your data).

The Cloud Term

Between 1966 -2007, the general school of thought grew to be :-

I.T. wasn't one thing. Many aspects of I.T. created little or no differential value and were simply a cost of doing business (Strassman, 90s)
There is a correlation between ubiquity of I.T. and its strategic value (differentiation). The more ubiquitous I.T. was, the less strategic value it created. (Nick Carr, '02)
I.T. activities could be categorised into rough groupings such as software, platform and infrastructure (the actual terms used have changed over time but this concept is pre-80's)
Certain I.T. activities would be provided through computer utilities as per other utility industries (Parkhill & McCarthy, 60's)
There were several forms that these computer utilities could take including public, private, government and all manner of combinations in between. (Douglas Parkhill, 1966)
We would see the formation of competitive marketplaces with switching and federation of providers.
These computer utilities had certain common characteristics including utility charging, economies of scale, elastic and "infinite" supply etc.(Douglas Parkhill, 1966)
Whilst all activities have a lifecycle which they evolve along through the process of commoditisation, the shift from "as a Product" to an "as a Service" world would require several factors (i.e the concept, the technology to achieve this, the suitability of activities for service provision and a change in business attitude.)

Back between '05'-'07, there was a pretty crystal clear idea of what was going to happen:-

A combination of factors (concept, suitability, technology and a change in business attitude) was going to drive those I.T. activities which were common, well defined and a cost of doing business from being provided "as products" to being provided "as services" through large computer utilities. The type of services offered would cover different elements of the computing stack, there would be many different forms of computer utility (public, private & government) and eventually we would see competitive marketplaces with easy switching and consumption across multiple providers.

In '05, James Duncan, myself and many others were starting to build Zimki - a computer utility for the provision of a JavaScript based "Platform as a Service" - for precisely these reasons. The concepts of federation, competitive markets, exchanges and brokerages for service provision of a commodity were well understood.

Unfortunately in late '07 / early '08, the term "Cloud" appeared and the entire industry seemed to go into a tailspin of confusion. During '08, the "Cloud" term became so prevalent that if you mentioned "computer utility" people would tell you that they weren't interested but could you please tell them about "this thing called cloud".

So, what is Cloud?

The best definition for cloud today is NIST's. Using five essential characteristics (include elasticity, measured service etc), four deployment models (private, public, government etc) and three services (application, platform, infrastructure) it nearly packages all the concepts of computer utility, the shift from product to services and the different categories of the computing stack into one overall term - "cloud".

In the process it wipes out all the historical context, trainwrecks the concept of a competitive marketplace with switching and federation, eliminates the principle idea of commoditisation and offers no explanation of why now. It's an awful mechanistic definition which only helps you call something a cloud without any understanding of why.

However, that said, NIST has done a grand job of trying to clean up the mess of 2008.

In that dreadful year, all these well understood concepts of computer utilities, competitive marketplaces, the lifecycle of activities, categorisation of the computing stack and commoditisation were put in a blender, spun at 30,000 rpm and the resultant mishmash was given the name "cloud". It was poured into our collective consciousness along with the endless blatherings of "cloudy" thought leaders over what it meant (I'm as guilty of this as many others)

To be brutal, whilst the fundamentals are sound (commoditisation, computer utilities, the change from products to services etc), the term "Cloud" was nothing more than a Complete Load Of Utter Drivel. It's a sorry tale of confusion and a meaningless, generic term forced upon a real and meaningful change.

My passionate dislike for the term is well known. It irks me that for such an important shift in our industry, I have to use such a term and then spend most of my time explaining the fundamental concepts behind what is going on, why this change is happening and undoing the various "cloud" myths that exist.

Being pragmatic, I'm fully aware that this term has enough momentum that it's going to stay. Shame.

Mystic Me 3.0

2010-01-13T23:02:00.011Z

It's time for a spot of bleary eyed crystal ball gazing.

Last year, my predictions were fairly reasonable with 7 hits covering the commercial release of PLED TVs to our beloved government economists saying that 2010 would be worse than expected.

The jury is still out on house prices [Update : the December 2009 figures showed the first annual increase in house prices - 2.5% - since May 2008.] whilst we await the land registry report but alas two of the predictions were wide of the mark. The FTSE 100 failed to drop below 3,500, only hitting 3, 512 - no cigar there then - and Yahoo wasn't sold.

So, with the usual added vagueness, looseness of terms and general get out clauses, yawn with delight for :-

Mystic Me Predictions for 2010.

The number of mergers & acquisitions in the cloud computing and open source industries will reach fever pitch, surpassing previous years.
The first examples of people trading on variability in cloud infrastructure prices and the early formation of brokerage concepts will appear.
There will be no let-up in end user confusion surrounding cloud computing as would be thought leaders will embark on an orgy of term redefinition. Expect lots and lots of heated debates on how cloud isn't cloud computing isn't utility computing.
The distorted creative destruction meme of modern society (i.e. "out with the old, in with the new") will get ahead of our desire to consume technology. Despite many predicting the death of the book, paperbacks will have a surprisingly good year.
RPI in the UK will rise sharply and the FTSE 100 will drop below 3,000 during the year. Judging on past performance, the MPC will keep interest rates low because they're barking mad.
Under howls of protest, banks will be given more taxpayers cash. This will be despite being bailed out, given free cash through quantitative easing and then splashing lots of dosh on bonuses. The tired arguments that "no-one saw this second crisis" coming and that we "can't let the banking system fail" will be trotted out to order.
Despite independent estate agent surveys suggesting that house prices have risen a gazillion percent in the last minute, Land Registry house prices will continue to drop in the U.K.
Environmental forecasters will be befuddled by Arctic summer ice disappearance exceeding the worst predictions of current climate models.
There will be legal attempts to claim and quantify ownership of social networks as company IP.
The new Doctor Who will be pants and the attempts to spice it up and make it more gritty will look rather sad.

The king was in his counting house ...

2009-12-30T11:15:00.005Z

... handing out our money.

This was the year that Mervyn King & Alistair Darling managed to spectacularly fritter away billions of taxpayers' money.

I was never opposed to lending money to banks but quantitative easing (a dishonest way of printing cash and giving it away in truckloads to the usual cronies) was disgraceful.

The tally to date is that the taxpayer has been exposed to £1 trillion of potential debt through cash injections, state guarantees, quantitative easing and other interventions. As a result of this, the taxpayer is expected to lose anywhere between ten to a hundred billion. All of this is to prop up an industry which generates at best £200 billion a year in tax.

Why is it that when the taxpayer acts as a lender of last resort, we have to make a loss into the bargain? When the hard up resort to loan sharks, you never hear tales of some financial wheeze where money is given away.

Of course, it's different because we couldn't let the banks fail despite no-one explaining why not. Still that doesn't mean we have to be 'soft' and being the lender of last resort should be a time of piracy. For some reason the city, unlike the poor, got let off the hook.

We could (and should) have demanded equity equal to any loans plus the loan capital plus punitive interest rates, but we didn't. Where's our pound of flesh and 2000% APR?

We could (and should) have invested heavily in social housing, bought out the building industry when it was on its knees and grew our state owned banks by providing liquidity into the economy. We didn't.

What did happen was that Meryvn & Darling were cheered by the financial giants like a pub landlord who has wiped the tab clean for his heaviest drinkers. Naturally, the taxpayer got lumbered with the bill and the underlying causes of the mess (huge debt, delusional valuation, excessive gambling, economic instability) have been unresolved.

Expect more bad news to come.

At least Darling has got a potential excuse in trying to mess things up for the Conservatives. If only some of the largesse had been spent on things that really matter, like combating global warming (which from the Copenhagen Accord laughingly only gets £60 billion a year by 2020).

A wasted opportunity but then that's how I feel about New Labour - a decade of disappointment. Whilst the noughties have been personally good for me, in general it failed to live up to the expectations. Unless of course you consider that WAGs, myspace house parties, wii fit, 4x4's, an endless war on terror, draconian legislation reducing civil liberties, excessive celebrities and a highly materialistic and self serving environment are the pinnacle of human nature.

To summarise the noughties, you'd have to say "nought for the environment, nought for social mobility and lots of noughts for bankers".

On a positive note, Doctor Who was utterly brilliant.

Where is Amazon heading?

2009-12-14T15:52:00.010Z

There is something that I've always found confusing about Amazon's cloud strategy.

The development of EC2 & S3 makes good sense given the suitability of these activities for large scale volume operations (an activity that Amazon, as a book-seller, specialises in).

The growth of an ecosystem around these core services and the provision of these services through APIs are ideal. The solving of some of the transitional educational barriers to cloud (such as persistency through EBS) seems spot on and ... well the list goes.

However, I've never quite understood why Amazon chooses to cannibalise its own ecosystem (the creation of a hadoop service when cloudera existed, the creation of autoscaling when many alternatives existed) rather than buying-out those different groups.

Given that, I can't be sure of where Amazon is going to head.

Other than the eventual need to move into a platform space. the moves towards a spot market could suggest that Amazon might attempt to set itself up as the computing exchange for basic infrastructure resources. To do this, it would also need to define itself as the industry standard (not just the defacto) probably through an IETF certification route and hence encourage other providers to adopt this standard.

Fortunately for Amazon, it already has several alternative open source implementations to support this claim and these open source technologies (such as Ubuntu Enterprise Cloud) provide a quick means for providers to get started.

There is huge future value in the exchange, brokerage and trading businesses for computing resources. It's what I was intending to go after with Zimki, all those years back.

I'm not going to make any predictions for now, I'll leave that until early January. However, if I was a betting man then I wouldn't be surprised if, over time, Amazon :-

Goes for IETF standardisation of EC2 & S3
Allows reserved instances to be traded on its spot market hence creating a basic form of commodity exchange
Enters the platform layer of the computing stack with provision of a development platform
Allows other providers (who adopt the EC2 / S3 standard) to sell instances on the exchange
Exits the infrastructure provision business by selling on the EC2 / S3 services (at a phenomenal premium) whilst keeping the exchange and any established brokerage business

Mystic Meg Epic Fail

2009-12-14T14:14:00.015Z

I hate predictions.

Don't get me wrong, I don't mind the "oh, it's already happening but I'll pretend it's new" type of predictions because you're guaranteed to look good.

I can happily quote that "the cloud market will grow", "standards, portability and interoperability will become increasingly important" and "the platform layer will be a major market" will full knowledge that these are safe bets.

Problem is, that these aren't really predictions and I've got a big mouth. Hence, I tend to make predictions which tend to explode rather nastily.

For example, back in 2002 I was predicting a financial meltdown in 2005 due to the massive growth in debt. Did it happen? Nope. I was out by a couple of years but that's the point of prediction, the when is vastly more important than the what.

That said, I can happily get the what wrong as well. Hence back in January 2009 when the FTSE was at 4,608, growing rapidly and many were talking about a rebound - I had to go and predict that it would drop to 3,500 within the year. Did it? Nope, it got close at 3,512 but never quite made it (back to the drawing board with my economic model again).

However, I'd be safe talking about cloud wouldn't I? Turns out that I get that wrong too. Hence back in 2007, I was predicting that "six years from now, you'll be seeing job adverts for computer resource brokers".

Earlier this year, I realised that prediction was going to be spectacularly wrong and happen much sooner. Eventually, I even admitted as much.

Adding salt to a fresh wound, is Amazon's announcement of a fully fledged spot market.

I suspect, it won't take long for someone to offer spread betting on the Amazon spot price or for some form of OTC derivative to mitigate against fluctuation in price and cover the risk of paying the full on demand price (because of failure to buy). Of course, this would work a lot better if users could resell reserved instances on the spot market providing the basis for a commodity exchange.

Opening up the spot market to the resell of instances between consumers will enable market pricing, making reserved instances more attractive. This will provide Amazon itself with future capacity planning information.

An alternative would be for users to resell reserved instances back to Amazon for sale on the spot market. However, this depends upon upon a quartet of objective, offers, availability and pricing.

For example, if revenue is the main objective, then there are scenarios (especially in the early days) where an increased revenue will be generated by selling a smaller number of instances at a higher spot price, leaving unfulfilled demand and capacity. It should be remembered that this is not market pricing but Amazon pricing.

Under a revenue objective, the conditions where it will be viable for Amazon to increase capacity on the spot market by the re-purchase of reserved instances (presuming Amazon isn't playing a double booking game with reserved instances, which are in essence a forward contract) will be limited.

It all depends upon this quartet and the only thing that I'm sure of, is that my prediction is out by a few years.

Ouch ... damn, how I hate predictions.

Cloud Camp Frankfurt

2009-12-11T17:54:00.003Z

A few months ago I provided an introductory talk on cloud computing at Cloud Camp Frankfurt. I was asked to be vendor neutral, so it is light on Ubuntu Enterprise Cloud.

They've put the video of my talk up, so I thought I'd provided some links. Please note, it is split into two parts.

Cloud Computing - Part I

Cloud Computing - Part II

There are more videos on the Cloud Camp Frankfurt site, they're worth watching as the event was a blast.

Old yet new ...

2009-12-07T18:04:00.004Z

I'm just comparing two of my talks, both on cloud computing and if anyone has time, I'd like some feedback.

The first is my recent talk from OSCON in 2009 covering "What is cloud computing and why IT matters", the second is my talk from OSCON in 2007 covering "Commoditisation of IT"

They both cover the same topic matter but with a different viewpoint (N.B. terms have changed since the 2007 talk but I'd like some feedback on style & content.)

Both are 15 minutes long but which was better and more importantly, why?

OSCON 2009: What is cloud computing and why IT matters

OSCON 2007: Commoditisation of IT

Private or Public clouds?

2009-12-07T12:28:00.003Z

There is ample evidence to suggest that many common and well defined activities in I.T. are shifting from a product to a service based economy. Naturally this change creates a broad range of risks including :-

the risk of doing nothing as competitors gain advantage from economies of scale through volume operations, utility charging, ability to focus on core activities and a faster speed to market through componentisation .
transitional risks including confusion, security of supply, trust in new providers, transparency and governance.
outsourcing risks including suitability, vendor lock-in, pricing competition, second sourcing options and loss of strategic control.

For any organisation, it is a case of balancing the risk of not using the cloud against the risks of using it. There appears to be two general schools of thought on this subject.

The first school states that whilst the outsourcing risks will be solved by the formation of competitive utility computing markets (with easy switching between providers) these markets do not exist today. Hence, whilst the movement to public clouds is considered inevitable (bar for the largest companies and governments), we're still in a time of transition. Private clouds can help solve many of these transitional risks whilst preparing for a future movement towards public cloud services.

The first school accepts that private clouds have a role, it emphasises the importance of standards, of reducing barriers to education and promotes a hybrid model of both public and private clouds. It encourages a compromise between economies of scale and transitional risks during this time of change.

I've been an advocate of this first school of thought for many years (since before 2006).

The second school of thought states that private clouds aren't cloud computing and advocates adoption of public clouds. It dismisses the transitional phase and talks of continuous innovation in the provision of what is fundamentally a commodity (commonplace, well defined and hence suitable for service provision). It is almost purist by nature, sometimes describing public clouds as true cloud computing and finding little distinction between private clouds and virtualisation platforms.

I don't subscribe to this second school of thought. Basic economic sense and risk management would suggest that in this time of transition, organisations will attempt to gain some of the benefits whilst mitigating the dangers of this phase. Hence, for the next few years I'd expect the cloud industry to be dominated by hybrid models.

After which I'd expect it to become more slanted towards public provision (as competitive markets form) but nevertheless hybrid models will continue to have a role.

Capital...

2009-12-06T23:28:00.003Z

Over the years, I've often discussed the ideas of physical and human (intellectual and social) capital within organisations. I thought I'd cover some old ground again.

Organisations simply exist between the intersection of a network of people and a mass of activities undertaken. Remove this and you're left with what an organisation really is, nothing bar any remaining residual capital.

The act of people interacting with activities creates several forms of capital, three of the most interesting are physical, intellectual and social. All of these forms of capital are susceptible to the ravages of commoditisation.

We've already experienced the effects of commoditisation on physical assets. For example, the news industry was once able to use physical assets (large and expensive printing machinery) to control the activity of publishing - not only what was published but whom. In days of past if you wanted to be a journalist your options were limited.

Today, the digitisation of content and the spread of the means of mass communication have changed the rules and commoditised this activity. The barriers to entry have been severely diminished and anyone can publish, This means news organisations have been forced to seek other means of differentiation, value and ultimately control.

Equally, many forms of intellectual capital has slowly been commoditised. Whereas in the past you needed direct access to a lawyer to help with the arcane knowledge of how to write a will, today you can download forms online.

All manner of knowledge has been neatly codified, commodified (given a value for access) and ultimately commoditised (become standardised, commonly available, relentlessly driven to a lower cost) through market forces.

Access to knowledge can (and has been) an important mechanism of control for some organisations. The commoditisation of such knowledge diminishes this means of control. As a journalist may find they are less dependent on a news company in order to publish, a budding lawyer may find it easier to access the knowledge they need without a law firm.

Obviously both types of organisations still provide the benefits created by the internal ecosystem of a network of people and a mass of different activities (i.e. rapid access to certain skills, specialists and supporting structures). However, both types of organisation will also have social capital - interactions, reputation and relationships with others.

Hence a journalist or lawyer my choose to work with one particular organisation because it can offer access to the right people, provides a prestigious network and has a high amount of social capital.

I mention this because many social network tools are currently busy codifying relationships between people. Furthermore, some are also trying to identify and provide measurable value in those networks (the act of commodification). Could this onslaught also lead to the commoditisation of business networks?

Will we see a future where we can buy and sell access to a social network? How will this impact organisations who depend upon their networks and use access as a means of control? Will companies also attempt to control and own this network more tightly?

These are just some of the questions I suspect we will be facing over the next few years.

FlashForward

2009-12-04T23:42:00.020Z

I rarely watch T.V. series, however FlashForward was recommended to me. I have to say it's outstanding.

The principle behind the story is fairly straightforward. The entire world experiences an event which causes everyone to see the "future", six months ahead.

Of course, it's not quite that simple. The "future" seen is that of an alternative world (from the many world principle) and therefore doesn't necessarily represent the character's current future but instead another timeline where the character may have made other choices in their past.

This creates a continuous drama of whether the future visions are right or wrong and an obvious illusion of choice where people try to change (or make happen) a future which is not necessarily theirs.

This paradox is illustrated with one particular character Dr. Olivia Benford. In the "current" world Olivia is married to a recovering alcoholic, Mark Benford. However, in her "future" vision she is with another of the characters Dr. Lloyd Simcoe. The paradox is that the "future" vision is that of another timeline, hence there are three obvious possibilities for this other timeline. Either :-

She is married to Mark (as per the "current" timeline) but has an affair with Lloyd
The interpretation of the vision is incorrect (i.e. Lloyd is just staying in the house for some other reason)
Her timeline is different in the other world i.e. she attended Harvard, met LLoyd and was never married to Mark (or divorced him due to his drinking etc).

This is the deliciousness of the series which has many examples of these paradoxes woven into the plot. You're kept guessing whether the visions being seen are a possible future of the current timeline or not.

All the time, the characters in the "current" world are making choices based upon the belief that their "future" visions are correct and in some cases they are even trying to make them happen (even when those visions aren't possible because of other past choices).

Of course, there is also the subplot of what caused the event, will it happen again and several other plot lines. I hope they don't mess this up like Lost which became a repetitive bore.

For the time being though, it is utterly brilliant.