Ecosystem wars

Back in 2005, I started to talk publicly about lifecycle concepts and the importance of ecosystems. In the next two posts, I'd like to draw a line in the sand and cover those basics of ecosystem one last final time.

I'll take the liberty of assuming notions such as lifecycle, organisational profile, componentisation and consumerization are universally understood and start with an explication of what a business is.

What is a Business?

A business is a living thing, comprising a network of people, a mass of different activities, and reserves of capital including financial, physical, human and social. It consumes, it produces, it grows and it dies. Like all organisms, any business exists within a number of ecosystems in which it competes and co-operates with others; it’s shaped by and shapes its environment, and hence needs to adapt constantly merely to survive.

People come and go, activities change, and hence all firms are in a constant state of flux. In any industrial ecosystem, new activities (innovations) are a consequence of competition and those that are useful will diffuse throughout the ecosystem becoming more of a commodity. This constant change creates a paradox, identified by Salaman and Storey:

”Survival requires efficient exploration of current competencies and ‘coherence, coordination and stability’; whereas innovation requires discovery and development of new competencies and this requires the loosening and replacement of these erstwhile virtues.”

These two extremes of survival (today and tomorrow) have diametrically opposite concerns, and the techniques, tactics and methods needed to manage each are entirely different. Those who manage organizations are therefore caught on the twin horns of a dilemma: how is it possible to be standardized and efficient as well as innovative and new, without prejudicing your survival - either today or tomorrow?

The effects of this on business can be seen in the constant restructuring to cope with new paradigms, and in the yo-yoing of popular management theories between opposites in a scramble to maintain order. A more effective balance can be found through embracing both goals simultaneously.

How do we balance both goals?

This requires a rethinking of how we organize, and a realization that what really matters is not innovation or efficiency per se, but how we continuously manage the path between the two. To explain why this is the case, let us examine a typical profile of an organisation (see figure 1) and consider how we build systems for the future.

Figure 1 - Using profile to build systems (click on image for higher resolution)

First, let us take those cost of doing business activities which act as underlying components of other business activities such as payroll, computer infrastructure, authentication etc. All of these activities are linear, well defined, ubiquitous and suitable for provision as standard re-usable components through utility services. Such an approach will increase our agility, rate of innovation (componentisation effects) and reduce the cost of gambling for any innovative activities built upon these utility services.

However, it is important to understand that all innovations (i.e. those activities which are uncertain and rare) are a gamble and whilst we can reduce costs we can never eliminate it. The future value of something is inversely proportional to the certainty we have over it, we cannot avoid this information barrier any more than we can reliably predict the future. However, there is a means to maximise our advantage.

By making these utility services accessible through APIs, we not only benefit ourselves but we can open up these components to a wider ecosystem. If we can encourage innovation in that wider ecosystem then we do not incur the cost of gambling & failure for those new activities. Unfortunately, we do not enjoy the rewards of their success either

.

Fortunately, the ecosystem provides an early warning mechanism of success i.e. adoption. Hence by creating a large enough ecosystem, we can not only encourage a rapid rate of innovation but also leverage that ecosystem to identify success and then either copy (a weak ecosystem approach) or acquire (a strong ecosystem approach) that activity. This is how we maximise our advantage.

To capitalise on this, we simply drive our newly acquired activity towards utility service provision and create the next wave of innovation through further componentisation effects. In this manner we create a virtuous circle of encouraging innovation in the ecosystem, leveraging the ecosystem to identify the next pattern and commoditising the pattern to utility services in order to encourage the next wave.

In effect, by being at the heart of this ecosystem we manage to create the simultaneous appearance of being highly innovative (as others in the ecosystem do this for us), highly customer focused (by leveraging the ecosystem to identify rapid diffusion) and highly efficient (as we focus on commodity provision). Tom Peter's old adage of choose one is a busted flush in this brave new world.

I've summarised these concepts in figure 2

Figure 2 - ILC model (click on image for higher resolution)

What is critically important to note, is that in essence both innovation of new activities and efficiency of commodity provision can be outsourced. The innovation of activities can be outsourced to a surrounding ecosystem building upon our services, whilst those commodity activities we consume can be outsourced to a marketplace of utility providers. There are benefits to retaining some element of control in those areas but the critical area to focus on is the transitional phase and how you leverage the ecosystem. Our ability to exploit the link between innovation and commodity depends upon the size, composition, engagement, speed of information and overall level of activity within that ecosystem.

Whilst each company comes with its own personal ecosystem (its staff, its partners) growing an extended ecosystem and using that to manage change is a powerful tool and a competitive weapon. It's not businesses but ecosystems that collide in the commercial world and woe betide that organisation which has the weaker. We've seen this story repeated many times before at many different levels through all the usual examples of BetaMax vs VHS or TCP/IP vs IPX/SPX.

It's imperative to understand that a well formed and large ecosystem leads to lower costs of innovation, creates higher rates of innovation, improves the rates of successful adoption, encourages greater efficiencies and can be mined to detect future successes. However, there are cost to this especially in the form of data collection which is why utility based ecosystems (where information is derived from consumption of APIs) have an advantage over product based ecosystems (where information is derived from market research). 

If it is just you and your employees then you'll find it difficult to survive a direct onslaught from any of the modern day monsters with well developed and extended API driven ecosystems (such as Google, Amazon etc). Not impossible, just damn difficult.

For more details, my OSCON '10 talk provides a useful overview of these concepts. However this is enough of a base to begin with.

In the next few posts, I'll cover some of the consequences of this and the strategies which are commonly deployed. Again, I realise this is nothing new and apologies to all readers who've had to endure me harping on about this over the last five years but I'm just scene setting for much later posts.

Consumerization

The evolution of any business activity is complicated by the existence of multiple ecosystems such as the enterprise and consumers. Activities can exist in, transfer between and commoditise within any of these ecosystem.

For example, whilst CRM has mainly commoditised in the enterprise space, email has crossed over to the consumer space in which it has been more aggressively commoditised. Today, these heavily commoditised mail services (such as Google Mail) create adoption pressure for the enterprise. These concepts are summarised in the figure below.

Figure 1 - Consumerization & Lifecycle Across Ecosystems (click on image for higher resolution)

When discussing an activity, a key issue to consider is in which ecosystem the evolution of that activity is governed i.e. is it being driven by competition in the enterprise or the consumer space. Back in 2001, Douglas Neal and John Taylor described the process by which governance transfers from the enterprise to the consumer ecosystem as Consumerization.

The consumerization of technology and its subsequent commoditisation in that consumer ecosystem can have a profound effect on the operation of any enterprise using that technology. It should be noted:

• Not everything transfers. Some activities find meaningful use within only one ecosystem and hence tend to develop only in that environment. For example, CRM.


• Barriers to entry can erode quickly. The impact on the media industry of the commoditisation of the means of mass communication occurred primarily in the consumer space over a single decade. Originally, the principal means of mass communication was provided through printing presses and supply & distribution chains. These acted as a major barrier to entry into those industries. The internet and digitization of content brought new means of mass communication into the consumer ecosystem and subsequently were aggressively commoditised resulting in widespread use of blogs, wiki and social media tools such as twitter. A consequence of this is that the barriers to entry into the media industry have been completely eroded. Anyone can become a digital newspaper.


• Ecosystems are not isolated. People are part of multiple ecosystems, hence activities in the enterprise (which may be provided through products) tend to start with custom-built solutions and hobbyist equivalents in the consumer space.


• The rate of commoditisation is not uniform between ecosystems.

Numerous activities have undergone this change. The pressure for adoption created through consumerisation has created positive responses such as BYOT (bring your own technology) and negative responses such as shadow IT i.e. when an organisation fails to deliver what its internal consumer need (Joe Baguley gave an excellent presentation on this topic).

All organizations should, as a matter of course, examine both their own and the consumer space for the growth of highly commoditised activities and consider the impacts of these and whether barriers to entry into their own industry will reduce.

For point of reference, computer utilities (i.e.Cloud computing) is expected to reduce the barriers of entry into many industries. In such cases an approach of adoption of the service and creation of higher-order systems can be used to re-establish those barriers.

Equally the impact of consumerization on the means of manufacture is going to hit a wide range of industries. This video is a foretaste of what is going to happen.

Building a private cloud

As we move from a world of infrastructure products to a competitive market of infrastructure services, private clouds have a valid role as part of a hybrid strategy in mitigating transitional risks (such as concerns over data governance, trust, transparency and security of supply).

The key word here is transitional i.e. this is not a permanent solution. Furthermore by using a hybrid strategy you're deliberately sacrificing some element of the benefits of public provision (economies of scale, focus etc) in order to mitigate those risks. It's a trade-off.

However, for this post I'll assume you've decided to make that trade-off. Hence you're going to be using a combination of public and private resources in a hybrid form for the immediate future. So dealing specifically with infrastructure, at what price should you be building a private cloud?

First, you need to recognise that a consequence of a shift to standardised components provided through utility services will be an explosion of innovation in higher order systems - i.e. the competitive landscape is going to become even more competitive. Second, cost efficiency won't result in reduced IT budgets because increased competition, the long tail of unmet demand and co-evolution effects will result in far greater consumption. Lastly, one of the key benefits of cloud is speed and when this is combined with cost efficiency, you'll end up doing more, faster with the same budget.

It's worth noting that there is little correlation between total IT spending and business value but there is strong correlation between speed and business value. How efficient your provision is in terms of speed is critical here. Now assuming you're not planning something daft like a long winded requisition process for new virtual machines, then the speed of provisioning will be minutes whether public or private. Assuming this is uniform, the distinguishing feature becomes cost efficiency - i.e. how much MORE you can do with same level of spending over competitors.

With public provision such as Amazon EC2, a small instance (I'll refer to this as a VM) costs currently $740 per year. Now whilst Amazon doesn't release figures, we believe the cost of provisioning such a VM can be much lower, possibly as low as $220 per VM per year though from the comments possibly much lower. Hence if your private cloud is being designed to cost $1500 per VM per year, you might be forgiven for thinking that this is only twice Amazon pricing and represent good value for a hybrid trade-off. What you're probably discounting is that your private environment will have to exist for three to five (in a worst case) years and during that time Amazon's pricing could fall rapidly, possibly to the order of $300 per VM per year, possibly lower.

In such circumstances, a competitor with a pure public play would have upto a 5x cost advantage (depending upon how much of your hybrid strategy was private provision) which is an astronomical difference in a world of increasing and more rapid competition. But that's a key point to note, the impact depends upon the type of industry you exist within and your competitor actions.

Assuming you're working in a high tech reliant industry, then in order to give your private cloud a fighting chance, you have to be designing for a target price of between $220-$540 per VM per year.

These figures (and lower) are achievable if done at reasonable scale and with a relentless focus on commodity provision. For reference purposes only, I've provided a hypothetical example (based upon real data from real examples) of a VM farm. I've used the following assumptions:

1. Base VM: 1.7 Gb RAM, 160 Gb HD, CPU equivalent to a passmark of 420
2. Target Price (VM per year): $510
3. Max Utilisation (to determine level of over-provisioning required): 70%
4. No. of VMs: 100,000
5. Base Unit: Rack
6. Focus: Commodity (No multiple PSU, Hot Swap)
7. Depreciation for hardware: 3 yrs
8. Current High Interest Rates of: 3%
9. Networks: All internal equipment and network upto external routers but not external bandwidth.
10. PUE:1.3

Given this, then figure 1 shows as a rough guide, how your target price per VM per year breaks down into its various components.

Figure 1 - Cost per VM per year.
(click on image for larger size)

Now obviously there are many ways to skin a cat, and pricing is sensitive to various factors such as scale, density of VMs, power consumption, number of units per rack, location etc etc. However, I put this here to start a discussion as I would like to know what's your cost per VM in your private cloud?

-- 12 Feb 2015

It's 2015 and companies are still commissioning / building private clouds as new projects! Ouch. There's going to be some very costly white elephants.

Componentisation

Herbert Simons showed in the Theory of Hierarchy how the evolution of a system is dependent upon the organisation of its subsystems. In short, as an activity becomes commoditised and provided as ever more standardised components, it not only allows for increasing speed of implementation but also rapid change, diversity and agility with systems that consume the activity as a subsystem.

In other words, the creation of standard sizes of bricks, pipes, utility services and other architectural building blocks led to a faster rate of house building and a wider diversity of housing shapes. It's the same with electronics and every other field you care to look at.

Commoditisation to standard components leads to an explosion of innovation for higher order systems.

The cloud is no different. It's the creation of good enough, standard components which will cause a wide diversity of higher order systems. These will in turn undergo their own evolution to more of a commodity. Without this process we would never have got to a CPU, let alone cloud or the hugely complex systems that will develop from it.

This also doesn't mean that innovation stops with the standard components. Whether it's brick making or electricity provision, there is a huge amount of operational innovation hidden behind the "standard" however the "standard" acts as an abstraction layer to this. Just because my electricity supplier has introduced new sources of power generation (wind turbine, geothermal etc) doesn't mean I wake up one morning to find that we're moving from 240V 50Hz to something else. If that constant operational innovation was not abstracted behind the standard then all the consumer electronics built upon this would need to continuously change - the entire system would collapse in a mess.

Whilst cloud computing is all about commodity provision of computing resources through utility services, these components act as the abstraction layer. We're not quite there yet, we're still in transition from a product to a utility service world. However, even in that utility service world, providers will still innovate in terms of operations but to a consumer that will be abstracted and ultimately reflected in better price and quality of service. To the consumer this will be seen as operational efficiency and the activity itself for all sense of purpose will remain a commodity, just like electricity and every other industry that has undergone this change. Of course, I'll do more things with it, create new and ever more complex higher order systems but my innovation is based upon the activity being provided as a commodity.

I can understand how many people don't relish the prospect that some activities within IT are becoming more of a commodity. They may not realise the benefits this will create in terms of progress but at the same time as they cry this isn't happening they happily consume systems which depend upon this process.

If you really don't like the idea of commoditisation and componentisation then try getting by without using anything which has depended upon it. Try building a computer without a CPU, electronics, electrical power supply or even nuts and bolts.

The online grocery guide to keeping me happy

There are many things which I find annoying about shopping online, the most irksome generally deal with grocery shopping. Now let's assume you're an online grocer and you've covered all the basics of having stock, delivering what you say you're going to deliver, the produce being in a reasonable condition and it arriving on time (a tall order for some).

Then, my advice to you is to apply what I call the Wardley Happiness Equation of Extreme Exceptionalism (wheee! for short). The equation is trivial to apply and will almost certainly guarantee my happiness with the shopping experience.

Wardley Happiness Equation of Extreme Exceptionalism

If I'm spending over £70 with you, then you can assume that this is part of a weekly shop. Hence when you come to tot up the final prices, simply calculate the delivery time plus 7 days and then for each item discount 15% for every day which the best before date is less than this.

Please don't ask me to check the dates or tell me that lettuce has a shelf life of three days - it's nonsense. There is nothing that annoys me more about online shopping than a week's worthy of grocery turning up with some best before dates of tomorrow.

The confusion of choice

I've been asked many times whether there are any mechanisms to slow down, prevent or reverse commoditisation? Well, there are a long list of techniques which can be employed and to explain them let's just revisit the lifecycle approach.

Figure 1 provides a visible representation of the evolution of an activity from innovation to commodity. It should be noted that what drives the evolution of an activity through its lifecycle is both user competition (the drive for something which is useful i.e. has utility) and supply competition (the drive to provide a better way of meeting users need over competitors).

Figure 1 - Lifecycle
(click on image for larger size)

Furthermore as an activity transitions from one phase to another such as custom built to products or products to utility services, a number of factors are required for the transition to occur. The factors include:-

• Concept : e.g. the idea of providing something in that format such as computing infrastructure through a utility.
• Attitude: a willingness for consumers to accept the activity in the new format.
• Technology : the technology to actually achieve this.
• Suitability : the activity to be widespread and well defined enough to be suitable for provision in that format e.g. a utility format requires a well defined and ubiquitous activity capable of supporting the volume operations needed.

So how do we counter this?

Don't Sell it to Everyone: One of the simplest ways of ensuring that something doesn't become a commodity is to not sell it to everyone i.e. prevent it becoming ubiquitous and well defined. Unfortunately competition applies here, so you have to find a way of preventing others from selling - i.e. patents give a time limited opportunity to do this.

Create Confusion: If you can't prevent everyone selling, then create a confusion of choice. The purpose of this is to reduce certainty over the activity in order to persuade consumers that it is more of an innovation. The most profitable scenario is to have every consumer buying it and believing it is highly innovative to them alone - the "designed for you" meme.

Examples of confusion of choice can be found from light-bulbs to mobile phone contracts. Walk into your average electrical department store, go to the light bulb counter and ask yourself - "Do I really need a choice between 200 lightbulbs and bewildering array of styles, plug formats, power outputs, swirly bits of glass, shapes etc? Didn't we use to have less choice and wasn't it easier to make a decision then?"

This abundance of choice is not designed for your convenience but to prevent you from making comparison. It's the same with the plethora of designed for you mobile phone contracts or insurance instruments or ... the list is long. Why else do you think we now have a vibrant price comparison industry countering this.

Association: one of my favourite examples of manipulation is to try and install a hidden meaning to an activity i.e. to create an association between this particular brand and either a lifestyle choice or some other implied benefit. If you're lucky, turning an ordinary activity into a status symbol can reap huge dividends. This sort of manipulation can be most easily recognised by switching on the TV and watching adverts - most of which rarely discuss the activity itself but imply a consequences of choosing this particular brand. Using this brand of soap will transform you into a glamorous person, whisked away to shower in a tropical forest surrounded by other glamorous people? The use of this deodorant will make you instantly appealing to huge numbers of the opposite sex? The purpose of these acts is not to cause confusion but to fundamentally add a hidden utility to a particular brand of the activity and to disassociate it from the rest.

Complexity: Another tactic is to reduce certainty through the introduction of visible engineering i.e. make the device look more complex than it is. It's actually a combination of confusion of choice, association and complexity which turned the humble vacuum cleaner into a space age construction. It doesn't matter that the activity is ubiquitous, well defined and commodity like nor that the latest range of cleaners may be no better at cleaning. Today, vacuums have new properties such as "doesn't lose suction", they use visible engineering and a bewildering array of choice and associations to lifestyle. All of this is designed to persuade consumers that there's more to a vacuum cleaner than cleaning the floor. Kerching!

Bundling: Another favourite of mine is to bundle many activities together. For example, the iPhone is not a phone but a wide range of activities bundled into a single device - calculator, phone, camera, calendar, alarm clock, browser, email, time recorder, map etc. Bundling provides ample opportunities for confusion and dissociation, i.e. you can't easily compare one smart phone to another and certainly not the pricing tariff of a standard phone to a smart phone. However, through Android, Google has aggressively commoditised most of these bundled activities with the use of open source and open ecosystems and hence cornered Apple into a very high risk, constant innovation game. So as well as tactics to counter commoditisation, you can also use counter tactics to force it and your competitors into tricky positions.

I've always been a big fan of Apple, however this is the second time in my opinion that they've been seriously outmaneuvered and the root causes of this appear to be the same - a poor understanding of change and how to play the game. They're great at innovation but that's only part of the battle. Time will of course tell, but I don't see Apple's future as rosy.

I won't go on as there's a whole mass of techniques and counter techniques in this space. However to the question can you slow or reverse commoditisation - well you can certainly persuade consumers of this.

Private vs Enterprise Clouds

There is a debate raging at the moment between different types of clouds hence I thought I'd stick my oar in. First, let's clear up some simple concepts:-

What is cloud?

Cloud computing is simply a term used to describe the evolution of a multitude of activities in IT from a product to a utility service world. These activities occur across the computing stack.

These activities are evolving because they've become widespread and well defined enough to be suitable for utility provision, the technology to achieve this utility provision exists, the concept of utility provision is widespread and there has been a sea-change in business attitude i.e. a willingness to accept these activities as being a cost of doing business and commodity-like.

There is nothing special about this evolution, it's bog standard and many activities in many industries have undergone this change. During this change, the activities are moving from one model to another which creates a specific set of risks (both real and perceived) around trust in the new providers, transparency, governance issues and concerns over security of supply. There are other risks (including outsourcing and disruption risks) but these are irrelevant for the specific question of private vs enterprise clouds. Nevertheless, all of these "risks" help increase inertia to the change. For clarity's sake, I've summarised this evolution in figure 1.

Figure 1 - Lifecycle and Risk.

(click on image for larger size)

Why private clouds?

Private clouds (where the service is dedicated i.e. "private" to a specific consumer) are generally used as part of a hybrid strategy which combines multiple public sources with a private source. The purpose of a hybrid strategy is simply to mitigate transitional risks (concerns over governance such as data governance, trust etc). This is a normal supply chain management tactic and occurs frequently with this type of evolution. Even within the electricity industry you can find a plethora of hybrid examples in the early days of formation of national grids.

Fundamentally, a hybrid strategy mitigates risk but incurs the costs of both lesser economies of scale and additional resource focus when compared to a pure public play. It's a simple trade-off between benefits and risks which can often be justified for a time.

Why Enterprise Clouds?

Enterprise clouds need a bit more explaining and to understand why they exist we first have to start with architecture. To keep things really simple, I'm going to focus on infrastructure.

As per above, the use of computing infrastructure has undergone a typical evolution from innovation to commodity forms and more recently the appearance of utility services. In the earlier stages of evolution, the solution to architectural problems was bound to the physical machine. Scaling was solved through buying "a bigger machine" i.e. Scale-Up whereas resilience involved hot-swap components, multiple PSUs and other redundant physical elements (the N+1 model). This was essential because of the long lead time for replacement of any physical machine i.e. there existed a high MTTR (mean time to recovery) for a physical server. Applications therefore developed on the assumption of ever bigger and ever more resilient machines. These architectures spread (specific sets of knowledge behave just like activities) and became best practice.

As computing infrastructure became commodity-like, novel system architectures such as Scale-Out (aka as horizontal scaling) developed. These solved scaling by distributing the application over many more smaller and standardized machines. The new architectural solution was therefore "buy more machines" and not "buy a bigger machine". This scale-out architecture rapidly spread as infrastructure became more generally accepted as a commodity.

As we entered the utility phase, infrastructure has in effect become code – that is, we can create and remove virtual infrastructure through API calls. The MTTR of a virtual machine provided through a utility service is inherently lower than its physical counterpart and novel architectures called "design for failure" have emerged that exploit this. The technique involves monitoring a distributed system and then simply adding new virtual machines when needed.

In the cloud world application scaling and resilience are solved with software whereas in legacy it was often solved by physical means. This is a huge difference and I've summarized these concepts in figure 2.

Figure 2 - Evolution of Architectures

(click on image for larger size)

By necessity, public cloud infrastructure is based upon volume operations – it is a utility business after all. Virtual compute resources come in a range of standard sizes for that provider and are based upon low cost commodity hardware. These virtual resources are typically less resilient than their top of the line physical enterprise class counterpart but naturally they are vastly cheaper.

It should be noted that a ‘design for failure’ approach can take advantage of these low cost components to create a far higher level of resilience at any given price point through software.

By way of example, the general rule of thumb is that each 9 roughly doubles the physical cost. Hence let's take a scenario of a base machine designed to give a 99% up-time with a machine designed to provide 99.9% uptime costing twice the amount etc.

In the above scenario, using four base machines in a distributed architecture provides a theoretical up-time of 99.999999% though naturally it will suffer periods of degraded performance due to single, dual or triple machine failure. In a utility computing environment, this impact is negligible due to the low MTTR of creating new virtual machines and in such as case you have a low cost environment (4x base unit), high level of resilience for total failure (1 in 100 million) and fast recovery for periods of degraded performance due to the low MTTR.

Now compare this to a single physical machine (ignoring all scaling, network and persistence issues etc). The equivalent physical machine would be 16x more costly (assuming the 2x rule holds) and in the rare situation that complete system failure occurred, the MTTR would be high. In reality, MTTR would be high for all its components unless spares were kept.

Now for reasons of brevity, I'm grossly simplifying the problem and taking a range of liberties with concepts but the principle of using vast numbers of cheap components and providing resilience in software is generally sound. We've seen many examples of this including RAID. These cloud architectures are no different except they extend the concept to the virtual machine itself.

The critical point to understand is that these two extremes of architecture have fundamentally different models based upon the differences in the underlying components i.e. the architectural practice co-evolved with the activity. The cloud model is based upon volume operation provision of commodity good enough components with application architectures exploiting this through scale-out and design for failure. Now in the following figure I've tried to highlight this difference by providing a greyscale comparison between a traditional data centre and a public cloud on a number of criteria.

Figure 3 - Data Centre vs Cloud.

(click on image for larger size)

OK, now we have the basics let's look at the concept of Enterprise Cloud. The principle origin of this idea is that whilst many Enterprises find utility pricing models desirable, there is a problem when shifting legacy environments to the cloud. Now when companies talk of shifting an ERP system to the cloud they often mean replacing own infrastructure with utility provided infrastructure. The problem is that those legacy environments often use architectures based upon principles of scale-up and N+1 and infrastructure provided on a utility basis doesn't confirm to these high levels of individual machine resilience. The problem is simply people are trying to shift applications built with best architectural practice for product based infrastructure to a world where infrastructure is a utility which has its own but different best practice. It's not that legacy is wrong, it's just that in this case legacy means built with best practice for a product world and that practice is no longer relevant.

At which point the company has two options; either re-architect to take advantage of the volume operations through scale-out and design for failure concepts or demand for higher level resilient virtual infrastructure i.e. try and make the new world act like the old world.

This is where Enterprise Cloud comes in. It's like cloud but the virtual infrastructure has higher levels of resilience but at a high cost when compared to using cheap components in a distributed fashion. So why Enterprise cloud? The core reason behind Enterprise Cloud is often to avoid any transitional costs of redesigning architecture i.e. it's principally about reducing disruption risks (including previous investment and political capital) by making the switch to a utility provider relatively painless. However, this ease comes at a hefty operational penalty. The real gotcha' is those transitional costs for redesign increase over time as the system itself becomes more interconnected and used.

In principle, Enterprise Clouds are used to minimise disruptional risks but they will be ultimately subject to a transition to the new architecture because of high operational costs. There are other reasons for an "Enterprise Class" cloud but most of these, such as where data resides, can also be provided by using a "Private" cloud that is built using commodity components until suitable "Public" clouds are available.

The different economic models is what separates out private / public compute utilities from enterprise clouds, I've highlighted this on the greyscale in figure 4. Actually, I prefer to use the original term virtual data centre rather than enterprise cloud because that's ultimately what we're talking about.

Figure 4 - Enterprise vs Private Cloud.

(click on image for larger size)

Do Enterprise Clouds have a future? Sort of, but they'll ultimately & quickly tend towards niche (specific classes of SLAs, network speeds, security requirements etc). Their role is principally in mitigating disruption risks but the transitional costs they seek to avoid can only be done at an increasing operational penalty. It should be noted that there is a specific tactic where an enterprise cloud can have a particularly beneficial role: the "sweating" of an existing legacy system prior to switching to a SaaS provider (i.e. "dumping" the legacy). In most cases, Enterprise clouds will become an expensive folly.

Do Private Clouds have a future? Yes, they have a medium term (i.e. next few years) benefit in mitigating transitional risks (such as issues over data governance) through the use of a commodity based model. However, be careful what you're building and remember the impact of private clouds will diminish as the public markets develop. I say "be careful what you're building" but in practice this means don't build. The vast majority of companies lack the skills and management capability necessary and though you're trying to build a "commodity" based private cloud, the chances are you'll end up with some very "enterprise cloud" like.

Do Public Clouds have a future? Absolutely, this is the long term economic model and will become the dominant mechanisms. Public utilities should also encourage clearing houses, exchanges and brokerages to form. 

Last thing, I said I would focus on infrastructure to keep things simple. The rules change when we move up the computing stack ... but that's another post, another day.


--- 8th February 2016

Note to self.

Five years later and ... oh, my past self would not want to know.

People are still building private clouds and trying to push enterprise class cloud despite the obviousness of the changes. The whole co-evolution of practice finally became firmly established as DevOps but has gone a little bit out of control in terms of making somewhat grand claims of cultural changes. The points of inertia and the overstating of risks continues in corporations but we're finally getting through that point in the punctuated equilibrium that this will all get washed away.

Overall, it has been a torrid journey. Many are still confused on basic concepts such as the change of practice with activities. We're still having this discussion today! I have to caution that legacy isn't flawed as much as built with best practice for a product world. Many are trying to "create" new paradigm shifts in order to re-establish flagging business models by "having another go". In many cases, the strategic play of many former IT giants has been next to hopeless. There's quite a list of well known names circling the spiral of death by cost cutting to restore profitability whilst the underlying revenues are in decline.

It has become plainly clear that the level of blindness to the environment (i.e. poor situational awareness) is incredibly high in most corporations at the executive level and far worse than I could have possibly anticipated. Inertia will always need to be managed but that such companies failed to manage predictable changes with so much warning is ... stunning. 

Will Cloud Computing help the business align to the market?

I was recently at a private conference of CIOs when in the midst of discussing activities such as CRM to ERP, it became clear that not only did everyone understand these applications, but also that every company had them. Almost all of those companies have expensive customization programmes in place, tailoring these systems to fit their needs. This behaviour is puzzling, but to explain why, we'll need to first look at the concept of business evolution.

All business activities share a common evolutionary pathway, a lifecycle; from the innovation of an activity, to custom built systems implementing that activity, to the provision of products, to eventually commodity provision and the appearance of utility services.

It is unusual to find activities that are commonplace, well understood, well defined and accepted as a cost of doing business being treated as though they were in an earlier stage of evolution with heavy customization. But each CIO told how that was exactly what was happening at their company, and the discussion became surreal when we discovered that many of the customizations being made to systems like CRM were common as well.

Surely, these activities are best served by being provided as a standardized commodity ideally through a market of utility services? Isn't that the whole point of cloud computing? So why aren't we all flocking to consume a host of common activities through the utility services of cloud providers? Well, a diverse range of companies already are.

However, several of the CIOs who had looked into the issue said these services didn't work for their company. On further questioning, it became clear that it wasn't IT but the business that was pushing for the customization in order to fit in with their way of working. Several CIOs had even used the standardized services of cloud providers to challenge this - asking the business to justify the additional orders of magnitude costs for a customized system by demonstrating the differential value that 'their way' made.

Whilst we often talk about business and IT alignment, this shouldn't mean IT just delivering what the business asks for. In circumstances where there is no differential value, then it's better to 'fit to the model' rather than 'fit the model to us'. We do the former all the time - from banking to electricity provision; we don't go and build our own customized solutions, we just find a way to work with the dominant market standards.

Will cloud computing help the business to treat common, cost-of-doing-business activities as though they are common and a cost of doing business? Or, will some businesses continue to spend vast sums customizing that which really makes no difference?

Could the cloud help the business re-align itself with the market?

Reprinted from LEF site

VMware as an acquisition target?

Back in 2009, I proposed that VMware (or more importantly its master EMC) would eventually divest itself of its virtualisation business. The reason for my thinking was as follows :-

1. Whilst the majority of VMware's revenue was based upon its virtualisation technology, this was an area that was ripe for disruption through two points of attack - open source systems such as KVM and the formation of marketplaces offering utility based virtual infrastructure. The latter almost certainly requires open source reference models to avoid issues around loss of strategic control and when combined with aggressive service competition around price, this doesn't leave much room for a license based proprietary technology.


2. A dominant position in the enterprise is no guarantee for future success - see Novell Netware and the IPX/SPX vs TCP/IP battle. Critical in such battles is the development of wide public ecosystems and inherently open source has a natural advantage. However, given the revenue position of VMware, it could not afford to undertake this route.


3. The obvious route for VMware would be to develop a platform play, most likely an open source route with an extensive range of value add services - from assurance to management. The current business would be used to fund the development of this approach until such time as the company could split into two parts - virtualisation & platform.


4. Given the likely growth of private clouds as a transitional model in the development of the cloud industry, VMware would position the virtualisation business in this space for a high value sale, benefiting from its strength in the enterprise. This is despite it being unlikely that VMware would become the defacto public standard, that the technology was likely to be disrupted, that hybrid clouds are a transitional strategy superseded by formation of competitive markets and that many "private clouds" would be little more than rebranded virtual data centres.


5. During this time, there would be signals of confusion over the VMware strategy precisely because it would be using a time limited cash cow to fund a new venture whilst preparing to jettison the cash cow prior to disruption.

Given the confusion over cloud, the often central (but ultimately misconceived) role that virtualisation is given in the industry, the generally disruptive effects of this change and the wealth of many competitors then in my opinion with luck, timing and good judgement a buyer could be found.

So, in my opinion:

• For VMware, it would mean creating a strong platform business funded by its current revenue stream before jettisoning the virtualisation business at high value prior to its disruption.


• For the buyer, it would mean ... whoops ... well that's capitalism for you. Next time, pay more attention.

Of course, this is just my opinion but I haven't changed my view over the years. I'm expecting to see an increasingly clear division within VMware between platform and virtualisation in the next year or so.

I'm hence curious to know what others think? Do you believe that VMware would sell its virtualisation business?

Should I listen to my users?

Following on from my previous post on lifecycle, I thought I'd discuss the question of when to listen to users. I'm going to use my lifecycle and profile graphs to illustrate the points I'll be raising, so if this is not familiar to you then please read my most recent post on lifecycle.

Figure 1 - Lifecycle (click on image for higher resolution)

Figure 2 - Profile (click on image for higher resolution)

Before I start, a word of apology. All of this stuff is blindingly obvious and has been for many many years - hence please don't take offense.

Any new activity starts in the chaotic phase and then as it evolves through its lifecycle it enters a more linear phase. As it moves, its characteristics change from uncertain, deviating and a source of worth to known, standard and a cost of doing business.

Less confusion creeps in, let's just reiterate that a new activity is an innovation. Whilst we tend to abuse the term innovation, a feature differentiation is a feature differentiation, a process improvement is a process improvement and an operational efficiency is an operational efficiency e.g. you can call every process improvement, operational efficiency and feature differentiation an innovation if you want but good luck in trying to make sense of things if you do.

As explained in earlier posts, as an activity's characteristics change then the methods by which you can effectively manage it change as well i.e. we shift from agile to six sigma for example. This is why there is no one size fits all.

Equally, in the chaotic stage the approaches taken are about gambling, experimentation, potential future worth and novel practice (i.e. it's highly uncertain), however when that same activity has entered the linear stage it's all about conformity to standards, defined processes, price vs quality of service and best practices (i.e it's predictable). In between these two extremes is where ROI (return on investment) matters because unlike the chaotic stage a market exists to provide some basis for trending and unlike the linear stage, you have a choice over whether to implement it since it is not a cost of doing business.

When it comes to users then :-

• In the chaotic stage the only reason why you would listen to potential users is the same reason why you might collaborate with others - serendipity i.e. the chance encounter of a better idea. Of course, whether the idea is better or not won't actually be known until you "experiment" i.e. you put it out into the market. You have as much chance as identifying the future successful innovation as any user and there are no methods of guaranteeing any innovation will be successful. Like it or not, you have to gamble. The rule of thumb is listen to yourself.


• In the transition phase listening to users is essential because a market has established, users are becoming familiar with the activity, customer feedback and trending is possible and competitors can be analyzed. The rule of thumb is listen to your ecosystem.


• In the linear stage, the activity is a commodity and its all about price vs QoS. Now, assuming you're not going to embark on a disinformation campaign and attempt to persuade users that a commodity is actually some form of innovation (a common tactic) then the only thing you need to really focus on is your position against competitors i.e. faster, more reliable, cheaper etc. Hence the rule of thumb is pricing & quality comparison.

So, should you listen to users? The answer is "yes and no", which one depends upon the stage of lifecycle of the activity in question.

[Added Note 26/02/11] I've just come across this Fred Wilson quote : "Early in a startup, product decisions should be hunch driven. Later on, product decisions should be data driven". It's pretty much spot on.

Deconstructing Gartner's Hype Cycle

This is a piece of work that I did many years ago but given my recent post on Lifecycle (nee evolution). I thought I'd revisit it. I will assume the reader is entirely familiar with the concepts of commoditisation.

In figure 1, I've taken part of the evolution curve and modeled onto it a differential benefit curve (differential value - cost of implementation). This latter curve shows how the benefit of an activity changes as it evolves from its early innovation (where it is a strain on company resources) to a late product stage where the activity is ubiquitous and of insignificant differential value between competitors.

Figure 1 - Lifecycle & Differential Benefit (click on image for higher resolution)

When a new activity appears, you often get whitepapers written about it. These are generally done at a time when the activity is showing a highly positive differential benefit. Obviously there is a delay between the collection of data, the publication of the whitepaper, a user reading the whitepaper, the decision to do something and then implementation of the activity.

By the time an activity is implemented, the actual differential benefit may be vastly different. This creates a delta for expectation i.e. a difference from what we thought we would get and what we got. Figure 2 provides a graphical notation of this.

Figure 2 - Delta in Expectation (click on image for higher resolution)

Modelling this delta, with an awful lot of assumptions, provides the expectation curve shown in figure 3. There are a complex set of assumptions and conditions around this, however for the purpose of this post then I'm happy to say this curve is somewhat valid ... caveat, caveat ... except where it's not :-)

For the sake of this post, let's just pretend it is. What the curve shows is the early stages start with low expectations (but possibly high hopes), expectations are then quickly exceeded and continuously rise to reach a plateau after which expectations rapidly become unfulfilled leading to a trough of disillusionment before eventually levelling.

Figure 3 - Expectation Curve (click on image for higher resolution)

This all sounds strangely familiar and so it should. The expectation curve quite neatly maps to Gartner's hype cycle - see figure 4. So, we have a potential basis for explaining the underlying forces behind the hype cycle except of course, we have all the assumptions and I'm talking about expectation and not visibility. I'm not even sure what visibility actually means but given it's a hype cycle, I'll assume high hopes (expectations) means high visibility. Clocking up another assumption here.

Figure 4 - Hype Cycle & Expectation Curve (click on image for higher resolution)

Does our new underlying "basis" of the hype cycle shed any new light on the subject? Well, yes. However before I show this, I'd like to examine the final stages of lifecycle.

The evolution of an activity from products to utility services invokes its own expectation curve not through differential value (the creation of a new activity) but operational efficiency (a more efficient means of providing an existing activity).

In figure 5, I've provided the later stages of lifecycle including the transition from products to utility services and modeled an operational benefit curve (operational efficiencies over competitors - cost) of a transition to utility services. Again, lots of assumptions.

Figure 5 - Lifecycle and Operational Benefit (click on image for higher resolution)

NB. this benefit curve is the same shape as the earlier differential curve being derived from the benefit created over competitors, number of competitors exploiting the change and a changing cost of implementation due to maturity. Hence the transition to utility services starts with a period of investment, a rapid benefit over competitors gained by those creating or exploiting such services and then a decline in benefit over competitors as more companies switch to a utility model.

The reason why I mention this, is that whilst Cloud Computing is all about volume operations for ubiquitous and well defined activities (i.e. use of computer resources in business) and is hence all about commodities, this transition will create a similar expectation curve around operational efficiency in much the same way that a genuine innovation creates an expectation curve around differential value. This is shown in figure 6, and the result is the same delta in expectation curve shown beforehand.

Figure 6 - Delta in Expectation (click on image for higher resolution)

Our first "insight" is Gartner's hype cycle tends to show both innovative activities and transitional effects on the same graph. We should be careful to distinguish between operational efficiency (doing the same thing better) and differential value (doing a new thing) lest we start to confuse Cloud Computing with Innovation.

Hence, in the following hype cycle I've highlighted several activities, including :-

• cloud computing: more efficient provision of the existing activity of "using computer resources in business"
• social network analysis: a relatively new activity and a potential differential

Figure 7 - Expectation and Hype Cycle (click on image for higher resolution)

Since we started with differential value or operational efficiency, we can now map "value" zones onto the hype cycle. I've done this below.

Figure 8 - Hype Cycle & Value Zones (click on image for higher resolution)

What this suggests is the early stages of the hype cycle has an increasing benefit. In the trough of disillusionment, there is still benefit to be gained but it's diminishing. However, in the slope of enlightenment and the plateau of productivity there is little or no differential or operational benefit over competitors since everyone else is doing it. That's our second "insight".

The lesson of this story has been known in military circles for a long time. An imperfect plan executed today is better than a perfect plan executed tomorrow i.e. if you wait until the activity can be easily and effectively implemented (the plateau of productivity), it'll provide little competitive benefit to you.

Fortune favours the brave.

[A final few comments]

To generate the expectation curve I had to create a model over time. This required lots of assumptions because the evolution (lifecycle) curve does not have a time axis (i.e. you can't predict when something will evolve). There are hence a couple of points I'd like to make clear.

1. You can't simply overlay the expectation curves of different activities on top of each other - i.e. the axis of time is different (some are stretched, some are shortened). Gartner's curve doesn't define its time axis and we can therefore assume they're referring to a general shape which appears over an undetermined length of time.
   
2. The Gartner curve specifically refers to the technology trigger. We can assume this is when the technology starts to spread and ignores any early stage effects (invention etc).
   
3. If the Gartner curve was based upon the measurement of some physical property, it would be possible to reverse the process i.e. from Gartner curve to expectation curve to evolution lifecyle and accurately state where an activity was along the uncertainty axis. By very definition this is impossible. I can currently only state where something was in the past once it has become a commodity. Hence I have to conclude that Gartner's curve is not based upon some external measurement of physical property but instead it is more likely by a process of expert review (i.e. averaging where forecasters think something is on the curve) or even more simply, analysts placing dots on the curve.
   
4. The Hype Cycle in its current form can have both novel activities and the evolution of the same activity to more commodity forms represented on the same position of the curve at different times i.e. a single activity may well go through the the peak of inflated expectations multiple times. In the first case, this will refer to the differential value of the novel activity but later on, the same activity will appear in the same position due to operational efficiency of a more commodity form. This occurs because the same activity can be given multiple different memes i.e. x86 architecture, client/server, hosting or cloud and whilst those memes are different, the underlying activity (i.e. provision of computing infrastructure) is the same. You cannot therefore use the Hype Cycle to determine evolution, notwithstanding the issue that it is time based and evolution cannot be measured over time (we have no crystal ball).
   
5. The expectation curve matches the Gartner curve in certain circumstances. I cannot conclude much about the Gartner curve other than to say that its shape appears to have some validity in specific circumstance and I can approximate the value zones where the curve does match. This doesn't say anything about where the dots are placed on the curve just that the rise of increasing then diminishing then plateau of negligible benefits seems broadly right. That up, down, almost flat shape seems to have merit.

-- Update 10th Feb 2015

This uses an old form of the evolution graph. I've subsequently refined the terms innovation, custom built, product and commodity to genesis, custom built, product (+rental), commodity (+utility). The problem was that whilst I used "innovation" to mean the first ever attempt to put an activity in practice, the common use of the word "innovation" (used liberally to mean genesis of something, feature differentiation of a product, introducing a utility model for an existing activity) made that meaningless. Hence, I changed to use "Genesis" to re-assert the point.

-- Update 7th Sept 2015

Gartner Hype cycles no longer use Visibility vs Time but instead Expectations vs Time. The "time" axis seems to be only a generic sense of travel as the items on the hype cycle are given times to reach the plateau e.g.

I have no idea why they've made this change. Obviously, I was using an expectation curve and so again I broadly agree with the shape. I'm somewhat concerned that the axis changed but the curve didn't but given that the dots are just aggregated opinion, I suspect the curve is just an opinion as well. This doesn't mean it's not useful, as long you keep in mind it's just opinion all the way down.

Pioneers, Town Planners and those missing Settlers.

All business activities evolve, they share a fairly common lifecycle described in the following diagram. From innovation, to custom built examples, to productisation (including the appearance of rental services) and finally to commodity (including utility services).

Figure 1 - Lifecycle (click on image for higher resolution)

As those activities evolve, their properties change from a chaotic to a linear extreme. In the chaotic stage, the activity :-

• deviates from what has existed before and is a novel practice.
• is dynamic and constantly changing.
• is rare and poorly understood.
• has high levels of uncertainty and it is not possible to predict future outcomes.
• has no market data, competitor analysis or well understood trends.
• has characteristics which emerge as we learn about it.
• is strongly affected by serendipity, chance encounters and discovery.
• is a potential source of future worth, differential and hence competitive advantage.
• is a gamble

By the linear stage, that same activity has evolved and:-

• is mature and rarely changes.
• is standardised with a wealth of best practice.
• is commonplace and well understood.
• has a high degree of certainty and known impacts.
• has an abundance of market data, competitor analysis and trends are well known.
• has well defined characteristics.
• has well defined procedures and plans for implementation.
• is a cost of doing business with little or no differential advantage except through operational efficiencies.
• is a known quantity.

Now all businesses consist of a mass of activities, each of which may be at different stages of their lifecycle (stage of evolution). You can map a single business by examining the components involved in a line of business and their stage of lifecycle. You can also examine broader effects by plotting the frequency of activities at different stages of lifecycle thereby creating a profile for an organisation or an industry. This is shown in the figure below, to which the chaotic, linear and in-between stage of transition has been added.

Figure 2 - Profile (click on image for higher resolution)

The techniques which you use to manage each of the phases of profile (chaotic, transition, linear) are entirely different because the fundamental characteristics are different. Which is why no one size fits all approach to management exists. For example, agile development approaches are ideal for the innovation (chaotic) and early transition phases but are superseded by more structured approaches such as six sigma in the late transition and commodity (linear) stages. You can't apply one size fits all without either hampering innovation or impacting efficiency. You need multiple techniques, multiple types of people and even multiple cultures. Alas we ignore it.

In many areas of management, this creates a constant yo-yo between one extreme approach and another such as : agile vs six sigma, networked vs hierarchical, push vs pull. The answer is invariably you need a balance of both. The trick is to learn when to use each.

Given all this, here are my questions :-

1. Since lifecycle is constant and the properties of activities change as they evolve through their lifecycle, why do we organise ourselves around type of activities (e.g. IT, Finance, Operations) especially as a "typed" approach leads to outsourcing of inappropriate activities, misapplied techniques & alignment issues between groups?

2. Why don't we organise ourselves instead by lifecycle with specialist groups managing each stage of lifecyle regardless of the type i.e. an organisation based upon Pioneers, Settlers and Town Planners?

3. Most companies have Research & Development groups (equivalent to Pioneers) and common or shared service groups (equivalent to Town Planners) but Settlers seem to be invisible. Why is this? Who manages the transition from innovation to commodity in your organisation?


-- Update 8th March 2015

A bit of digging in the old memory banks, brings me to Robert X. Cringely's book, Accidental Empires reissued in 1996, page 235 - 238. Copying some quotes from that book (which I recommend people go buy and read), the ideas of pioneers, settlers and town planners are all there. I knew it had come from somewhere.

Think of the growth of a company as a military operation, which isn't a stretch, given that both enterprises involve strategy, tactics, supply line, communication, alliances and manpower.

Whether invading countries or markets, the first wave of troops to see battle are the commandos. Commando's parachute behind enemy lines or quietly crawl ashore at night. Speed is what commandos live for. They work hard, fast, and cheap, though often with a low level of professionalism, which is okay, too, because professionalism is expensive. Their job is to do lots of damage with surprise and teamwork, establishing a beachhead before the enemy is even aware they exist. They make creativity a destructive art.

[Referring to software business] But what they build, while it may look like a product and work like a product, usually isn't a product because it still has bugs and major failings that are beneath the notice of commando types. Or maybe it works fine but can't be produced profitably without extensive redesign. Commandos are useless for this type of work. They get bored.

It's easy to dismiss the commandos. After all, most of business and warfare is conventional. But without commandos you'd never get on the beach at all. Grouping offshore as the commandos do their work is the second wave of soldiers, the infantry. These are the people who hit the beach en masse an slog out the early victory, building the start given by the commandos. The second wave troops take the prototype, test it, refine it, make it manufacturable, write the manuals, market it, and ideally produce a profit.  Because there are so many more of these soldiers and their duties are so varied, they require and infrastructure of rules and procedures for getting things done - all the stuff that commandos hate. For just this reason, soldiers of the second wave, while they can work with the first wave, generally don't trust them, though the commands don't even notice this fact, since by this time they are bored and already looking for the door. While the commandos make success possible, it's the infantry that makes success happen.

What happens then is that the commandos and the infantry advance into new territories, performing their same jobs again. There is still a need for a military presence in the territory. These third wave troops hate change. They aren't troops at all but police. They want to fuel growth not by planning more invasions and landing on more beaches but by adding people and building economies and empires of scale.

Robert X. Cringely, Accidental Empires, 1996 (the reissued, I don't own the original).

What Robert called Commandos, Infantry and Police is what I later called Pioneers, Settlers and Town Planners. The two are identical except Robert was there much, much earlier. In fact 1993, a good ten years before I implemented the tri-modal structure,

I owe Robert Cringely a debt of thanks and hence the update.

-- Update 4th May 2016

These days I use terms such as uncharted to describe the more chaotic and industrialised to describe the more linear.