Thursday, January 16, 2014

Mapping the competitive landscape

A business is a living thing.  It consists of a network of people, a mass of different activities, practices and data along with reserves of capital including financial, physical, human and social.  It consumes, it produces, it grows and it dies.  We often attempt to understand a business through the use of box and wire diagrams whether business process maps or IT systems diagrams or … choose your poison. However, these diagrams whilst great at making connections between things do little to help us determine user needs, how things change and how we should manage them.

To make matters worse, the box and wire diagrams often have implicit information embedded in the terms and acronyms we use.  This makes them unintelligible to the uninitiated. Try asking someone from the business to decipher a diagram full of labels such as ERPM, BIM, PIMS and task management modules.  Or ask your long suffering IT folk to make sense of Ship Call Reports, Energy Balance, GAP adjustments or Peaking Inventory Projections.  Alas, within most organisations you have disparate groups with their own worldview based upon box and wire diagrams with their own terminology.  Is it any wonder that discussions between those groups are rife with alignment and translation issues?

To emphasise this point, I’m going to use a simple box and wire diagram (figure 1) for an unspecified process from which I have removed the terms - only some of you would have known what they meant anyway.  I’ll ask a few basic questions of the reader.

Figure 1 – A box and wire diagram of an unknown process



Questions 
  1. Which components in the above diagram identify user needs and whatever is produced?
  2. Component E was outsourced to a third party resulting in a high cost overrun and poor delivery. 
    • Can you give an explanation as to why?
    • Would you recommend the outsourcing of component F?
  3. It has been recommended that A, B, C and F use a cloud provider. Do you agree?
  4. It has been recommended we adopt Six Sigma throughout the project. Do you agree?
I’ll now convert the above diagram into a map.  This is simply done by first arranging the items in order of a value chain from the visible user needs to the invisible components consumed and them mapping each component on how evolved it is.  See Figure 2.

Figure 2 – A Map view of an unknown process



From the above it becomes far easier to answer the above questions.

Answers
  1. Components B and C are high up the value chain and likely to represent the actual user need and what is produced from the process.
  2. On outsourcing
    • Component E is at an early state of evolution and hence is unlikely to be suitable for outsourcing as no economies of scale or standardisation exists. 
    • Component F being more of a commodity is highly suitable for outsourcing.
  3. Component A, C and F being more of a commodity are likely to be suitable for provision by a cloud provider. Component B is not.
  4. The process consists of multiple components at different stages of evolution and hence a one size fits all method is inappropriate. In this case: -
    • Components B & E are likely to be suitable for Agile, In-House development.
    • Components A, C and F are likely to be suitable for highly structured methods such as Six Sigma and use of outsourcing.
    • Component D is suitable for using an off the shelf product.
The diagrams (the box and wire versus the map) are of the same process with no labels but the mapping view provides far more situational awareness and enables conversations on what matters and how things should be treated. Of course, adding labels adds further context as more data is always better.

A case in point is when you add data on competitors' value chains. For example figure 3 is a comparison between one company (let us say you, represented by vertical blue lines) and competitors (represented by horizontal red ones).

Figure 3 - Box and Wire comparison of competitors


Now at best, from the diagram above you can see that you're doing something called component B which the competitors are not. Maybe it's a differential, maybe it's a wasted activity? It is difficult to tell.

Now if you map both and compare (see figure 4) then a different story emerges.

Figure 4 - Mapping comparison of competitors



From the above, it is clear that component B is something novel and likely to be a potential differential. But equally there is differences in components D, E, F.  In component D you seem to be using a product whereas competitors are custom building systems - this is likely to be an area of efficiency for you. But in components E and F then you seem to be behind the game and likely to be creating a source of inefficiency.

With a mapping view, I can :-
  1. Determine what is likely to be suitable for outsourcing or insourcing
  2. Determine what should be built with agile, or bought off the shelf or driven by a six sigma approach
  3. Identify areas of efficiency, inefficiency and differential and what is suitable for cloud.
The above however is simply standard housekeeping stuff and the real power of mapping isn’t alignment and management methods but in understanding economic patterns and strategic gameplay. Mapping provides a mechanism for common understanding in an organisation and learning of such patterns.

I've been using mapping for over eight years (since the first maps of Fotango) with exceptional success and during that time I've discovered and tested many different economic and strategic patterns. At its heart, mapping simply provides a view of the chess board - you still have to learn how to play the game and unfortunately I can't condense eight years of experience into a single blog post.

However, I do run one day workshops for the members of LEF (a private research group of large companies) and occasionally I talk about these methods in public.  An hour long video covering some of the topics can be found here and also, for those willing for a more deeper dive then this uncompleted (but useful) series of post will help.  You can always come and find me at OSCON (the greatest conference in the world) which is the one place I almost always attend or alternatively you can search through this blog.

To give you an idea of the scope of mapping, the workshops cover such themes as -
... beyond this, it boils down to experience and developing lots of it. However, that's the point of this post, mapping is not a discrete thing but a journey just like learning to play a game of chess. Of course,  as with chess it always helps to start by looking at the board first! Also, as with chess then I'm afraid you have to learn to play the game and you can't farm this off to someone else.

Strategic gameplay is complex and it doesn't fit into a nice 2x2 diagram. There is a great wealth of patterns to be learnt, understood and applied if you can embrace the complexity of what is competition or as I like to call it, organisational warfare.  As with any competitive engagement, maps and situational awareness are critical to the outcome.

Once you start to get to grips with this, how to manage an environment, how to exploit it, how to deal with uncertainty and the arsenal of tools you can deploy then you can learn to play a good game.

Oh, and before you say 'it's an IT' thing and that you work in business ... well, I developed the technique when I was CEO of a Canon subsidiary to cope with competition and the world of change that I was experiencing.  I happen to use examples of IT companies because they generally seem to play a better game. These days I use the techniques in a wide variety of industries and governments from giants to start-ups.

As a rule of thumb, many organisations I meet have two apparent problems which threaten their future livelihood. The first problem is called 'IT' and it often suffers from a range of difficulties from single size methods to excessive reliance on outsourcing to alignment issues. The second and usually more fatal problem is called 'The Business' and it often suffers from a range of difficulties from inertia to poor situational awareness to lack of strategic gameplay to inability to identify value. Alas, there seems to exist a tendency to always point the finger at 'IT'.

In reality the problem is the division itself, an artificial construct of how we organise by type. In the past, I've found the way of solving this problem was to get rid of the construct but that's another post for another day.

Wednesday, January 15, 2014

One to remember

In my mind, without doubt, I've just witnessed the most enlightened tweet from Neelie Kroes, the EC commissioner for Digital Agenda.
This is exactly what Neelie Kroes should be doing,  reducing barriers to competition, promoting Net Neutrality in the EU and using this as a differential with the US.

Whilst we're at it, since the US is still debating over copyright on APIs then there's a number of other things that can be done.

1) Reinforce the concept that APIs are principles and not expressions and hence are not subject to copyright.

2)  Remove any vestiges of software patents (often as disguised business process patents) throughout the EU and encourage reverse engineering due to the fast and sequential rate of innovation in the industry. Copyright protection is more than enough in this field.

The problem of privacy.

The problem of privacy can be usefully summed up with the line

perceived value of usefulness >  perceived value of any immediate privacy / control concerns

Hence for example when we have events such as Snowden and revelations of secret laws then our general view starts to drift towards privacy / control concerns and we start to question whether the usefulness of services such as Google exceed this?  Prior to Snowden then usefulness generally outstripped any privacy / control concerns for the majority.

The real problem of course here is marketing and perception.  Whether we like it or not, marketing is very effective at promoting perceived value of usefulness and diminishing other concerns when it comes to selling stuff.

Post Snowden, marketing will continuously tip that balance in favour of selling more products 'tailored to us' and collecting more of our information.  Our concerns will diminish, they're doing it for our benefits and our needs - aren't they?


The system which basically intrudes into highly personal details of our lives is promoted as creating values of safety, security, comfort i.e. 'Mother knows best' etc.  You could do a mock up of the entire page with malicious hackers having taken control of your information which would make people run a mile.  But marketing won't do this, it's not their job to provide a balanced view, it's their job to promote a specific view and even create user needs when none existed before.

The problem you're fighting with privacy is therefore Marketing, our own perceived desires (whether real or fabricated for us) and the desire of companies to sell products and gain more access to our lives. Snowden was a fortuitous event that exposed some of the excesses of what is happening but it will be forgotten and ultimately buried. There will be niches, people with longer term memories who are not easily swayed by the bright lights of marketing - but they're niches.

If you want to fight the erosion of privacy and create an environment where users are under more control in a more egalitarian society then you either need to create commercial interests in line with this or robust legislation.  You need to be willing to use marketing to diminish the perceived value of usefulness and raise the perceived value of privacy / control concerns. 

This means getting dirty. 

You'd have to oversteer by promoting web giants as being 'Big Brother', 'friends of criminals' and at the same you have to rebrand the 'Department of NO' (i.e. security) as being 'Mother', 'Keeping you safe' etc.  To some extent this is already done but the balance of power is very much with the selling side of consumer services and most users are easily swayed to accept the loss of privacy in return for baubles of use.

I'm not saying you shouldn't be concerned about the loss of privacy and control - you should.  But in all likelihood privacy will be a notion of the past.  You're in an uphill struggle to prevent what is already evidently happening. Privacy and control concerns are being given away freely through choice in favour of more usefulness even when many of those useful needs are simply figments of marketing's imagination and didn't exist before marketing persuaded us we needed them.

You could take a 'pragmatic' approach of accepting the change and looking at how to minimise any risks but remember the competitive pressures push in one direction, so don't examine today's situation but work from the basis of tomorrow's.  In other words, start by considering a world where all user information, even what we would consider today as your 'private' details are not only a matter of public knowledge and discussion but accepted as such - every place you've been, every conversation you had, everything you ever looked at and every detail of your life.

Imagine a world where offers that are publicly promoted to you are influenced by the fact that six years ago you spent 2 seconds looking at a picture in some shop window (rather than the average of 1.5 seconds) and your heart rate increased slightly. Every minute detail of your life will be catalogued and referred to ... apparently, for your benefit.

Now ask yourself what needs to be put in place so that users retain some element of control?

That said, I must now get back to spying on my family with my 'mother' system - it's in their own good you know, the brochure tells me so ...

Friday, January 10, 2014

Cloud Recap + five and a bit years

tl;dr when faced with a system which claims to be PaaS then ask yourself - can I just write code, build data and consume services? If you need to do anything else other than this then it's not PaaS.

Five and a bit years ago I wrote a post called 'Cloud Recap', a summary of the previous years and the state of play in 2008. In this post, I'd like to revisit that earlier topic and also a concern I raised in 2009 on how 'Platform is being used at all layers of the stack, so we hear of cloud platforms for building IaaS and many other mangled uses of the concepts'

In the early days ('05-'07) back when we had systems like BungeeLabs, Zimki etc then we had a simpler division of the stack though somewhat misguided by the tendency to create too many layers. The key was always the division of responsibility i.e. with different layers of the stack, part of the solution became someone else's problem. Of course that created new problems related to second sourcing options and buyer / supplier relationship and hence portability was going to be an issue.

Since then the terms have changed, hardware is now infrastructure and frameworks are now platforms. The latter change is unfortunate because Framework as a Service clearly spelt out that you would be coding (and adding data, consuming services) in a framework whereas Platform as a Service could end up meaning a multitude of things - a coding platform, a deployment platform etc. This is what has happened and hence my concern back in '09.

Now to explain the difference and where I consider there to be a problem and divergence from the original path, I'm going to explain today's layers of the stack in terms of the old view (which I happen to think is cleaner). 

I'll first take a purist view. So, in reverse order, from figure 1

Figure 1 - An overview of the stack, old and new




The lowest layer of the stack is related to provision of virtual hardware either through virtual machines or containers such as LXC. Along with the evolution of activity (computing evolving from product to utility) then we expected to see a co-evolution of practice (as in architectural practice). We've seen this with the shift from scale-up to scale-out, N+1 to design for failure, disaster recovery to chaos engines etc. This has given rise to the 'DevOps' movement and hence there is a host of tools that have developed around configuration management, auto scaling, auto deployment, policies etc from Chef to Juju. This has unsurprisingly extended to concepts such as an application store with entire images, configuration and policy information bundled together. This low level of the stack is known as IaaS.

The second layer of the stack is related to simply the use of code, data and services. The underlying components including the management layers are de-coupled from the perspective of the user. The user is only concerned with writing of code, data and consumption of services. These sorts of concepts are encapsulated in systems like GAE, Heroku and Cloud Foundry. Obviously behind the scenes is a wealth of configuration management, auto-scaling, use of VMs / containers etc. This layer of the stack is known as PaaS.

The third layer of the stack (application) is related to provision of entire applications and application services e.g. salesforce. This remains relatively intact.

From a purist point of view, each of these layers would eventually be built on the other and each higher layer would provide increasing speeds of creation of higher order systems and agility at the cost of decreasing flexibility. Hence you would also get application stores of SaaS application built on PaaS along with PaaS built on IaaS.

Flexibility vs speed of higher order systems creation is the inevitable trade-off that componentisation creates (a good examination on this is provided by Herbert Simon).  Now key to this and the ideas of componentisation is minimisation of lower orders to good enough standards. So, for example in the IaaS space you would expect to see a limited range of virtual machine types (as you do with AWS). In the platform space, you would expect to see a limited range of coding environments (e.g. buildpacks).

However, there has been a divergence from this path and in particular with the idea of Application Containers. In these environments then applications are described in discrete virtual machines or containers along with configuration and policy information. Those application deployment environments and related app stores which manage these images are now unfortunately called PaaS.

Now, there's nothing wrong with application stores containing VM images or Containers or Cartridges for an application or an application component, offering autoscaling and configuration management with one click install and there have been many efforts to provide effective management of this (e.g. early CohesiveFT, JuJu, OpenShift Cartridges etc). But to encourage development of higher order systems on a plethora of different base components is foolhardy and likely to create a sprawling mess in the future.  Now, I'm not having a pop at Docker here because I happen to like Linux containers but I'm far from convinced that describing containers and the management of them as some form of development platform is wise.

You might have specific containers for specific languages and your code and data will be bound to it - which is all good.  But it's the limitation of choice which is key for effective componentisation.  Creating a platform on solely the idea of configuration of containers might give you freedom and flexibility but it would be the equivalent of enabling a multitude of perl compilers for different purposes. There is a significant long term cost in terms of management, portability and ultimately agility as more effort becomes focused on using the right container rather than simply coding. This is why I happen to like buildpacks and for the record, a limited number of buildpacks as the basis for a platform.

In the past, there was a very VERY hard line between what used to be called Frameworks and the underlying Hardware components. In a framework, I would write code, build data and consume services and that's all. If my framework had to ask me what environment I should use then it's not a framework. This is how the original Platform as a Service environments appeared to be designed, unfortunately the term has now morphed to include all sorts of things.

I raise this because of the Gartner MQ series on Enterprise Platforms which contains many that I would describe as true 'platforms' but also others which can be best described as deployment, configuration, autoscaling and image  / container management environments. The latter have a role to play, certainly for single click deploy of applications but if you intend to use these as development platforms then I would caution you to think carefully about sprawl before you end up having to build another system to cope with a mass of different base components.

Flexibility has a cost, it is not your friend. Such choice won't benefit development any more than being able to choose from a million different types of bricks would help housebuilding.

PaaS is all about writing code, building data and consuming services ... nothing else. It's the embodiment of what it known as 'NoOps' (a fairly awful and misleading term but then as an industry we're good at this - cloud etc).  This doesn't mean that there are literally no ops but instead ops is hidden behind the interface.  In a PaaS world, the developer doesn't care about VM's, containers, auto scaling and configuration between devices any more than they care about what hardware is used. The developer shouldn't even normally care about buildpacks just as long as one which will run their code exists on the platform. All the developer cares about is their code.

Naturally there is lots of misunderstanding and inertia around these concepts, that's normal but NoOps is going to happen whether people like it or not. Get used to it. Of course, that won't stop vendors marketing endless configuration, autoscaling and deployment systems as PaaS. Certainly, all these sub components are part of PaaS but they should be invisible.

So when faced with a system which claims to be PaaS then ask yourself - can I just write code, build data and consume services? If you need to do anything else other than this then it's not PaaS.

In the future, will I be able to sue my local burger bar for irresponsibly selling me a burger?

Location services are not new but the technology around this subject has certainly developed and improved over the last decade. In particular I take an interest in iBeacon. Now beyond the rather mundane from virtual post-it notes, development of the attention economy, in place transactions, enhancements to intelligent software agents to nefarious uses and what is called MALT (micromapping advertising location transaction) then there is a number of interesting possibilities.

For example, if my wearable happens to know how much I've been exercising, my heart rate, my cholesterol intake and transfers that information to the local burger bar as I walk in then not only can they change the menu and grass me up to my local health insurer for an unhealthy lifestyle but will they have a legal responsibility or duty of care to make sure they deliver healthy food? Will I be able to sue for failure to do this?

Will this encourage the formation of a black market of burger bars, off the grid with dampening technology preventing anyone snooping on my bad habits? Oh, this should be fun. The combination of wearable tech and transactional location based services has endless interesting possibilities.

Thursday, January 09, 2014

On mapping - some help needed.

I spend a great deal of time teaching people how to map the commercial landscape and exploit this to their advantage. An example map is provided in figure 1.

Figure 1 - Map from a large engineering company.


The process of mapping I've explained elsewhere but in principle - you start from user needs then you explain the value chain that meets those needs in terms of components and then you determine how evolved each of those components are. Maps are incredibly useful for determining how you should manage something, finding areas of inefficiency and strategic gameplay but that's not the point of this post (for more details on mapping the LEF has a video etc).

The axis at the bottom of a map is evolution and it describes a common pathway for how any activity, practice or data evolves from an uncharted and highly uncertain space to a more industrialised form. The path of evolution is described in figure 2.

Figure 2 - Evolution


Now, evolution is not the same as diffusion (a time based examination of adoption) but instead derived from supply and demand side competition. It cannot be measured over time due to an uncertainty barrier known as the 'future' and the impossibility of predicting individual actors actions.

However, there is a link between diffusion and evolution. For example, some actor will create a new activity, others will copy with custom built examples that diffuse, others will introduce products with diffusion of continually improving versions in the market until some actor introduces a commodity variety that will then diffuse. In each case adoption will be through a larger market and each variant (improved product etc) will increase certainty (i.e. maturity, feature completeness) of the act. 

A single line of evolution can consist of hundreds if not thousands of diffused examples of that activity and that evolution can take considerable time. For example if you consider the evolution of electricity, the nut and bolt and computing then you have a common pathway but over significantly different time scales :-

Electricity 
The history of electrical power generation can be traced from its genesis with the Parthian battery (around 200AD) to custom-built examples of generators such as the Hippolyte Pixii (1832) to the first products such as Siemens Generators (1866) to Westinghouse’s utility provision of AC electricity (1886) and the subsequent standardisation of electricity provision from the introduction of the first standard plugs and sockets to standards for transmission and the formation of national grids (UK National Grid, 1926).

Nut and Bolt
The genesis of the humble screw can be traced back to Archytas of Tarentum (400 BC). The principle was later refined by Archimedes and also used to construct devices to raise water.  Over the next two thousand years most screws (and any associated bolts) were cut by hand however demand for screw threads and fasteners created increasing pressure for a more industrialised process.  J and W Wyatt had patented such a concept in 1760 and Jesse Ramsden in 1770 introduced the first  form of screw cutting lathe.  However without a practical means of achieving industrialisation and with no standards then the industry continued primarily as was.  Maudslay then introduced the first industrially practical screw-cutting lathe in 1800 that combined elements such as the slide rest, change gears and lead-screw to achieve the effect.  However, whilst screws and bolts could be manufactured with inter-changeable components, the lack of any standards thwarted general inter-changeability. In 1841, James Whitworth collected a large number of samples from British manufacturers and proposed a set of standards including the angle of thread and threads per inches. The proposals became standard practice in 1860 and a highly standardised and industrialised sector developed that we recognise today.

Computing
The history of modern computing infrastructure can be traced from its genesis with the Z3 computer (1943) to custom built examples such as LEO or Lyons Electronic Office (1949) to the first products such as IBM 650 (1953) to rental services such as Tymshare (1964) to commodity provision of computing infrastructure and more recently utility provision with Amazon EC2 (2006).

The vast differences in timescale are in fact related to the evolution of the means of communication but that is a different story to this post.

I first described the pattern of evolution above at a talk at EuroFoo in 2004. I then collected a set of examples in 2005. However, it wasn't until 2007 that I was able to collect around 4,096 data points to propose that evolution was something more than just a nice idea and could actually be described as a weak hypothesis.

The pattern of evolution is based upon the properties of the component in question and the type of publications that surround it.  Table 1 provides a description of this and the terms I use for different classes of components such as activities, practices and data. For completeness I've also added the old terms I used to use.

Table 1 - Properties of Evolution


Now, it should be noted that when mapping an environment the components not only evolve due to competition but can also co-evolve. Most common examples of this are practices with activities. For example, best practice in a product world is not the same as best practice in a utility world.

The use of this pattern of evolution has been essential for me in not only mapping and strategic gameplay but determining the fundamental patterns of economic change, how organisations evolve, mechanisms for exploiting ecosystem, how to use open as a competitive weapon and the cycles of our economy. However, again this is not the purpose of the post.

So what is the point of this post? 

In table 1, under the class of data I have two labels - un-modelled for the uncharted and modelled for the industrialised. I have not been able to determine suitable labels for the transitional phases. You can see the properties above and examples for practice and activity class.

Hence, I'm looking for suggestions for the two labels.

Tuesday, January 07, 2014

Getting the property market moving

A lunchtime noodle ...

There is over 220,000 long term vacant properties in the UK along with apparently over 500,000 plots of land with planning permission that remain unfinished. They may have started, the proverbial dig a hole, pour some concrete and get building control to sign it off as a major start but they remain incomplete.

On the one hand we have a housing shortage, low levels of social house building, low levels of construction output, large amounts of land not being completed and still relatively high (though improving) unemployment in those industries whilst on the other hand we have concerns over house price inflation and high profits for building companies. Something isn't quite right here.

We need to fix this but we don't want to burden taxpayers given the time of austerity. Ideally, we want to turn it into a money spinner. So, here's a very rough suggestion.

1) Introduce legislation preventing pension schemes from investing in property, construction or land based funds. Whenever there is a problem, pension schemes often seem to be a likely dumping ground and so we want to protect them.

2) Introduce legislation to redefine any land with planning permission that is uncompleted in three years or any land which could have planning permission but not applied for in six years or any residential property which has less than six months main occupancy in a three year period as derelict. The period of time should not take into consideration whether the land has changed ownership through private sale or public auction.

3) Any land which is defined as derelict should be put up for compulsory government auction with no minimum price. The proceeds of which should go to the original owner minus a 40% administration fee  of the selling price payable to the Government for running the auction. Yes, the Government should run the auction and take a hefty cut and yes, the Government should also be able to bid. Any land which is unsold is valued at £0 and immediately purchased by Government. Any land which is sold through the Government auction should have the planning permission reset to the date of sale and the clock should start ticking again. 

4) Create a building fund from direct investment, unsold properties and from proceeds of the government auction for the purchasing of land for future social housing. 

5) Create a government owned national building company ideally through nationalisation of an existing building company after pressure has been applied to reduce its value i.e. encouraging short selling, any other dirty tricks you can think of etc. Allow the national building company free access to the building fund, invest in its growth through hiring and training and compete on the open market with an emphasis on social housing.

6) Allow and encourage local authorities to build up social housing through the national building company.

7) Make a declaration that the Government will consider privatising the national building company once four million new houses including at least one million social houses have been built or brought back into circulation in the marketplace.

8) Introduce a land value tax as an additional means of raising revenue.

Now, I'm certainly not saying you should the run the scheme exactly as I've written it above but what I want to point out is that the Government really needs to become a bit of a pirate here. The market is clearly not working as it should and we shouldn't be soft touches about dealing with the problem. In a time of austerity the Government needs to be a ruthless operator. I'm also not convinced by the compulsory purchase ideas that abound and I'm far more in favour of compulsory auctions with the Government taking a hefty cut in administration fees. But mostly I'm convinced by giving the building industry a kick up the backside and so simply threatening to do this is probably enough if it gets the results.

... whilst we're at is, can we please finally change the House of Lords to a House of Representatives selected by random from the electoral register to serve for a period of five to ten years.

Saturday, January 04, 2014

How to spot a benevolent dictator

There are many different types of ecosystems (consumer, provider, co-creation, two factor market etc) and the dynamics of any ecosystem are never static (unless it is dead). There are also numerous forms that a specific ecosystem can take, for example a free-for-all. a collective prisoner dilemma and a co-operative collective.

In the open source world, the most successful ecosystems tend to share a common key factor in the role of the benevolent dictator. So, I thought I'd write down a few lines on how you spot the benevolent dictator.

Who are they? A benevolent dictatorship can be provided by any form of entity. It might be a single person, a small clique, a company or even an organisation set up to govern the project. However, these entities have common characteristics.

First, they lead. They set the vision, the purpose and the technical direction. This can be achieved by an individual or a committee or even by use of community dynamics. For example, the technical direction could be determined by setting up incubation projects with a decision that those successful community projects will become part of the core. Such, incubation projects are a relatively effective means of identifying user needs. The key point to note is the benevolent dictator determines the means by which this technical direction is set.

Second, they set boundaries.  This can be through technical policies or ways of operating and governance including structures, hierarchies, appointment of roles, elections and order of succession etc. The benevolent dictator exercises control over these.

Third, they are willing and have the ability to say 'no'. For example, if the technical direction is chosen by an individual acting as the benevolent dictator then that individual must be able to refuse other alternatives. Hence, even if community dynamics are used then a 'successful' incubation project can still be rejected / cancelled by the benevolent dictator because the project is considered harmful to the overall ecosystem e.g. the development approach might be considered incompatible. Absolute authority derives from the ability to say 'no' and to exclude all other choices.

Lastly, they demonstrate benevolence. This doesn't mean that exploitation of the ecosystem doesn't occur but instead the long term overall health of the ecosystem is prioritised over shorter term commercial concerns and self interest. This is particularly difficult when a company acts as the benevolent dictator for a project which represents change to an existing business model of that company i.e. it has inertia to the change due to past business success. In such circumstances the company may tend to push the project in a direction of supporting its existing business models rather than focusing on end user needs and hence this is likely to endanger the long term health of the project.

If a project doesn't have an entity that shows those characteristics of leadership, setting boundaries, willingness plus ability to say 'no' and prioritisation of the long term overall health of the ecosystem over short term commercial concerns and self interest then it doesn't have a benevolent dictator.

It might have a dictator that isn't benevolent (which is not normally healthy long term as this tends to result in forking of the ecosystem) or it might have a benevolent committee which can't dictate (again not normally healthy long term as this to tends to create a collective prisoner dilemma or free-for-all). 

In my experience, I've yet to see a successful, purposeful and healthy ecosystem which doesn't have at the heart of it a recognisable benevolent dictator.

---- additional notes

1. For reference, privately I prefer to describe the benevolent dictator as the gardener of the ecosystem because the acts of shaping, nurturing, maintaining and harvesting are analogous. I also find that groups don't like being described as the benevolent dictator despite their characteristics pointing to them having that role. However, since benevolent dictator is in common use then I stick to that term to avoid confusion.

2. Absolute authority is derived from the ability to exclude choices. For example, suppose you have the right to say 'yes' but not 'no'. Then the right is simply one of blessing certain approaches, nothing can be excluded and at best you aim to seek a consensus. In such cases, anyone can do anything just some actions are blessed.  However, let us take the other extreme and assume you have the right to say 'no' but not 'yes'. Under such conditions you can exclude all other choices / actions bar the one you determine is the correct approach. The ability to say 'no' and to exclude is the bedrock of absolute authority and is essential to any form of dictatorship. Without it, you are forced to rely on creating a general consensus with the knowledge that anyone can do anything regardless and you are powerless to prevent it.

3. Political systems are the interplay of government, legislature and economic systems. Most of the systems we describe as meritocratic or democratic are in essence republics where members elect representatives to rule, mandate and enforce. The ability to do so depends upon the ability of the representatives to say 'no' to the members. If the representatives cannot dictate then you are likely to form a free for all with just the hope of an emerging consensus but no means of enforcement. Election does not also guarantee that the representatives will be benevolent, you merely hope they will be which is why it's important that matters of governance should be transparent. If you're lucky then your representatives will act as a benevolent dictatorship for the period of their rule. If you're unlucky then they'll either fail to be benevolent or fail to prevent a free for all or some other disadvantageous ecosystem forming.

4. There is always the hope that a continual consensus will form that no-one will break, no enforcement is required and no-one will ever have to say 'no' to anything. I've yet to see this successfully work in practice at scale. I would love to find a successful example where this is the case.

Thursday, December 19, 2013

A question for zero hour contracts

By zero hour contract, I don't mean you turn up for work, do basically nothing and go home with £300 as disgraced Lord Hanningfield claims is quite common in the House of Lords.  Zero hour contracts are ones where the employer has no obligation to give you work i.e. you can turn up for work, get told there and then whether you're needed and can be sent home without pay for the day.

Some groups such as the Chartered Institute of Personnel and Development have claimed that Zero hour contract workers are happy whilst others such as the TUC demand tougher regulation of these contracts. To quote the TUC

"The growth of zero hours contracts is one of the reasons why so many hard-working people are fearful for their jobs and struggling to make ends meet, in spite of the recovery"

Into this mix Vince Cable acknowledge the clear signs of abuse but decided not to ban them on the grounds they offer 'welcome flexibility'.

So are they good or bad? Well, in my view it's a form of exploitation which should not be welcomed but that's not the point of this post. Whenever you hear someone talking positively about zero hour contracts or providing surveys and market research proclaiming the benefit then the one question you should ask is :-

"If zero hour contracts are so good, I assume you're on one?"

If they're not, then show them the hand and walk away.

I, would of course welcome the use of zero hour contracts in the House of Lords. We could take a quick online vote as to whether we needed them today or not. Clearly, if Hanningfield's claims turn out to be true then many are just turning up to claim expenses.

I'll finally note that Hanningfield's defence of his actions were "I have to live, don't I?"  - well, there's always income support and housing benefit which is what the rest of the country uses. If you need the money Hanningfield then get on your bike and get a job. From the sounds of it there's a lot of highly flexible and 'happy worker' zero hour contracts out there.

Cloud Standards and Governments

As any activity evolves from product (+rental) to more commodity (+ utility services) due to the forces of competition then a number of factors come into play. There is a period of re-organisation as a new market emerges, a co-evolution of practice due to the changing characteristics of the evolving activity along with inertia caused by past practices and past means of operation. We see this with abundance in the cloud market today. 

Under normal circumstances a set of defacto standards would appear in the market and a period of consolidation then occurs.  Ideally, one common standard emerges but in some cases we get a limited set of alternatives. As the market continues to develop then multiple implementations of the defacto standards tend to form and a competitive market is created. 

It doesn't always happen like this, sometimes competition is restricted by the development of a monopoly / oligopoly. In such circumstances when user needs are not met or competition is restricted in a mature market then it may become necessary for Governments to force a set of standards onto that market in order to create a level playing field e.g. document formats.

There are also variations in the type of market that is formed. For a free competitive market (as opposed to a constrained or captured market) then the means of implementing the standards must be open to all. This is why the ability to reverse engineer an implementation is often critical to forming a competitive market. In the most ideal form the standard would be implemented in an open source reference model.

IaaS however is not a mature market but an emerging market for which we already have a defacto standard being formed in the AWS APIs. The rapid growth of AWS is testament to its meeting of user needs but nevertheless for reasons of buyer / supplier relationship and second sourcing options then a competitive market of many providers is desirable.  Fortunately, for the core parts of the AWS APIs (there is a long tail of functionality) there are already multiple open source implementations from Eucalyptus to CloudStack to Open Nebula and even parts of OpenStack. APIs themselves are also principles in US / EU law despite the best efforts of Oracle. Hence we have the basic pieces that are necessary to form a competitive market of AWS clones and possibly also GCE clones.

For an emerging market this is a reasonable position to be in at this stage. If anything Governments should push forward to consolidating the market by reaffirming APIs as principles and even limiting any disguised software patents that may limit re-use. This would all be positive.

However, vendors have inertia to change particularly when that change threatens their business.  So today, I increasingly hear calls from vendors for 'Open Standards' in IaaS on the grounds of meeting user needs for interoperability.  However,  what I normally hear from users is they'd like a number of alternative providers of the AWS APIs i.e. homogeneity in the interface but heterogeneous providers. There is a conflict here because the 'Open Standards' proposed are not AWS APIs whilst users rarely say they need another set of APIs other than AWS.  However, this difference in needs isn't going to stop vendors trying to pull a fast one by trying to ban competitor's interfaces through Government imposed 'Open Standards'. 

So when your local vendor comes and starts talking to you about the importance of 'Open Standards' ask yourself some basic questions ...

1) Is this an emerging market?
2) Is there a defacto standard developing in the market?
3) Is there one or more open source reference implementations of the defacto?

If the answer to all three is yes (as is the case with IaaS) then there is no need to force a standard onto the market and everything seems to be healthy. A good response to the vendor would be to tell them to come back with a service offering that provides the defacto.

If the vendor insists that you need their 'open standard', then ask yourself ...

1) Is the vendor 'open standard' different from the defacto?
2) Does the vendor's products / services offer the 'open standard'?
3) Is the vendor trying to create an advantage for themselves regardless of actual user needs because they cannot compete effectively in the market without this?

In all likelihood, the call for 'open standards' in an emerging market is just an old vendor play of 'if you can't beat them, try and ban them'. 

Caveat Emptor.

Tuesday, December 17, 2013

On Politics, IT and Strawmen

Shadow Minister Chinyelu Susan 'Chi' Onwurah describes how project management in Gov IT has been politicised. The article states a coalition portrayal of 'Waterfall as monolithic Labour' and 'Agile as dynamic, entrepreneurial Tory' and how to 'think that the solution to effective ICT deployment is to simply change your project management methodology is arrogant and naive'. It then goes on to describes how Labour will undertake this 'transformation with more humility' requiring 'diverse methodologies' and learning from 'the process itself'. 

This all sounds very reasonable but there is a problem with this portrayal. The UK GOV IT approach is already on the path to becoming more balanced. 

With any course correction, an over steer is a natural part of shifting an organisation away from one all encompassing method to a more balanced approach. I already see signs of the shift to a balanced approach happening which is remarkable in such a short period of time.  For example, whilst HS2 might be seen as controversial in some circles, it is showing signs of strategic understanding and IT leadership which I've rarely seen rivalled in the private sector. Rather than being 'one size fits all' shop with little of poor understanding of why action is taken, the CIO has been using a mapping technique to break down large scale projects and determine what methods are appropriate. Its future appears neither as an Agile / XP / Scrum Development nor a Six Sigma / Prince / Waterfall shop. Its future appears more balanced, more the use of the right methods where appropriate.

Unfortunately this doesn't help the argument that Labour will make a difference because it's difficult to claim a difference (a more balanced view) when this is already happening. The portrayal proposed in the article is unfortunately a necessary instrument of what we know as a strawman argument. If you were going to have two 'extremes' of past and present then in my opinion they would be :-

Under previous administrations, UK Gov IT could be characterised by an over reliance on large outsourcing arrangements, inefficiency in contracts, poor data and focus on user needs, poor development on internal engineering skill, extensive use of single and highly structured methods of project management, lots of discussion on open source / transparency / open standards but little execution.

Under the current administration, UK Gov IT can be characterised by a movement towards both insourcing where appropriate and use of SMEs, a greater challenge in contracts, a focus on user needs and collection of transaction data, co-operation with other governments, a focus on recruitment of engineering talent, use of more appropriate management methods from scheduling systems such as Kanban to a plurality of development techniques, a focus and action on open source, open data, transparency and open standards.

The change from these two states is not due to some political ideology but a desire of many people (both internal and external) to make things better and to reduce some of the excesses. Onwurah claims pointedly that Mark Thompson who responded to the Shadow Minister's speech also wrote the Conservative Technology Manifesto - something which I'm not aware of and a bit surprised about. I happened to work with Mark on a 'Better for Less' paper which had some input into the changes and that paper was driven solely by a focus on this principle of making things better.

For the record, until the worst excesses of spin of the Blair years, I had always voted Labour. Due to my disillusionment with Labour, I have subsequently voted Liberal Democrats. I have never once in my life voted Conservative. I don't personally agree with many of their policies. But then I wasn't asked to be involved in writing the paper because of my political beliefs but because the group felt I had something to contribute.  Politics never came into it.

But neither manifestos nor politics really made this change possible. As Mark pointed out 'the important point is that one would be hard pressed to find a single outcome in Labour’s Digital Britain to which those in the Government Digital Service (GDS), or their digital leaders network, are not already deeply committed'. The real change, something familiar to all of us that have worked in the open source world, was strong leadership and the UK GOV IT appears to have had this with Francis Maude.

I do fear that the worst excesses of sharp suited management consultants are itching to come back with the panacea of  “big IT” to achieve joined-up government. In my view, we would be wise to keep off that path and continue to create the more balanced future that seems to have already started. 

I have had concerns in the recent past that Labour has yet to show the example of strong IT leadership that will be needed going forward.  If it was up to me then I'd keep Francis Maude moving the changes forward regardless of political persuasion.  Maybe Shadow Minister Chi Onwurah can provide this future leadership - I simply do not know.  What I do know, from a competition viewpoint, is IT is more important to our future industry than petty debates and if this is what the future leadership might mean - politicising the debate and creating strawman arguments - then my concern grows. 

I hope I'm misunderstanding what is happening here but the politicisation of project management methods helps no-one.

Friday, December 06, 2013

Once more unto the breach, dear friends, once more ...

If you've been paying attention then you'll have noticed that the UK Government has had a fairly torrid time of trying to implement a policy for open standards.

It eventually won that fight despite efforts from numerous groups such as the BSA, large US companies apparently attempting to influence the processquestions raised over the role of one of the chairman  and software patent heavyweights piling into public meetings. At the end of several rounds of shenanigans and lobbying, the UK Gov was finally able to announce a comprehensive policy towards open standards. It was finally able to do what the only respected peer reviewed academic paper on the subject said was a good thing to do - adopt the policy.

However, this fight was just to get the policy approved. We should of course expect many more fights when it comes to specific open standards. Why? Well, it has all to do with money or more importantly the ability of certain companies to help themselves to large amounts of your and my money. 

To explain why, let us first understand why we need standards. One of the fundamental purposes of standards is to provide you with choice. If two systems use the same standards then you can switch between both and this switching encourages competition. The reason why UK Gov has a preference for open standards is that it also has a preference for a fair and competitive marketplaces i.e. if the standard is owned by someone or encumbered with patents then that person can limit competition in the market. 

Shouldn't we just therefore make everything into a standard? No. There is a cost with standardisation in that it limits innovation at the interface of the standard, hence the implementation of standards is only suitable for activities that have become commonplace and mature. 

But surely the marketplace as it matures will define its own standards? Yes. These are called the defacto standards and in most cases they should be adopted. However, in a mature market where the defacto doesn't meet some user need or encourage competition then Governments might need to introduce standardisation to create a competitive market.

Can you give me examples? Certainly. Two examples spring to mind.

1) In the case of cloud computing you have an evolving market where defacto standards are developing. As an evolving market it is too soon for any Government to attempt to implement standards. 

2) In the case of documents and word processing systems (like Open Office, Libre Office and Microsoft word) then you have a highly mature and established market. However, in this case despite there being a defacto there is not easy switching between a competitive market of products and this is most clearly shown when people talk of the cost of transfer from the defacto to another. 

So why should I care as a vendor? If you are the defacto provider for a mature marketplace and there is a significant cost for people to transfer off your system then you've in effect created a captured market and this can be very lucrative. You won't experience the normal competitive pressures of a free market, you can charge high prices knowing that most won't wish to incur a cost of transfer. This actually creates an incentive for you to lock people further into your system and you'll try and stop anything which threatens this such as open standards. 

So why should I care as a customer? Well, if it's a mature market with a defacto and there are limited alternatives unless you incur a high cost of transfer then you're likely to be paying well over the odds. What's operating is not a free market but a captured one and your exit cost from the existing defacto is simply a long term liability which will keep on increasing. In such cases, you should be in favour of open standards because that's the only way you'll get a fair and competitive (known as 'free') market and limit future liabilities.

So that gives some basic background on open standards. The real question is why do I bother mentioning this now? Well the UK Government has just launched a process to select open standards for government documents. I fully expect the lobbying machine to be out in full on this one as what's at stake is oodles of cash and control of a market.

Microsoft Office is obviously the defacto in a well established and mature market which has little differential between product offerings but also high costs of transition. The market has all the appearance of a captured market with weak competition. 

However, doesn't Microsoft Office 2010 uses .docx which is Microsoft Office Open XML Format and therefore an open standard? Alas no. 

Microsoft Office 2010 provides a number of formats (docx, pptx, xlsx) that collectively are called Office Open XML format (OOXML or OpenXML). This file format was submitted to ISO after intensive lobbying including accusations of rigging and it known as the OpenXML standard (ISO/IEC 29500 adopted in 2007). 

Alas, this standard was broken into two parts. There is the standard itself, which is known as strict OpenXML and was accepted by ISO (international standards organisation) and then there is transitional OpenXML which was supposed to be a transitional file format for Microsoft to ease the removal of some of the past closed source legacy from their file format. Of course, Microsoft Office 2010 implemented transitional OpenXML but only reading of strict OpenXML.

So when you use Microsoft Office 2010 to send a document in an OpenXML format such as .docx to someone who uses Libre Office which reads and writes strict OpenXML (as defined by ISO) and they edit the document and send you back a .docx file, then what you see is often a corrupted file or something which can only be read and has to be converted.

You probably think that something is wrong with their software. Well, this problem isn't because of their word processor but instead Microsoft Office 2010 didn't implement strict OpenXML except for reading.  It certainly claims that .docx is OpenXML, and that OpenXML is an open standard approved by ISO but the version of OpenXML that ISO approved (strict) is not the same version of OpenXML that Microsoft Office 2010 .docx uses (transitional). This means every other system has to reverse engineer the transitional OpenXML format that the default .docx uses and that's never perfect.

So let me be crystal clear, Microsoft Office 2010 didn't even fully implement the open standard it created and was approved by ISO after its own intensive lobbying. It did promise to provide reading / writing of this format in the future.

Hang on that's old news .... doesn't Microsoft Office 2013 now implement reading / writing of strict OpenXML? Well, the latest release of Microsoft Office does now implement strict OpenXML, unfortunately the default .docx is still transitional OpenXML, you have to specifically select strict OpenXML when you save the file (which is buried in the options) and of course, in order to use Microsoft Office 2013 you need to be running Windows 7 or Windows 8.

So basically, yes you can have the open ISO standard strict OpenXML if you upgrade your operating system, buy the latest version of Microsoft Office 2013 and remember to save in the right format. It must be noted that whilst Microsoft Office 2013 does now provide support for strict OpenXML and even includes group policies for this, the default is .docx which is using Microsoft's own version of an open standard (transitional OpenXML) but without actually being that open standard (strict OpenXML). It's all a bit messy and confusing.

So what does this mean in practice? Well, let us assume that the UK Gov chooses the ISO approved strict OpenXML as an open document standard. First, that means Gov departments would need to upgrade to Microsoft Office 2013 which means upgrading to Windows 7 or 8 and all the changes needed to applications (NB. Microsoft XP is out of support in April 2014). They'd also have to set group policies so that the default was strict OpenXML.

However, let us suppose you create a document, a .docx file and sent it to someone else who happens to be using Microsoft Office 2010. Unfortunately though they could read the .docx file (Microsoft Office 2010 provides read support for strict OpenXML) they couldn't write / edit such a document without changing it to another type of .docx (the default transitional OpenXML). Hence you might send out strict OpenXML but you could easily get back transitional OpenXML though both files are called .docx.

Now given that some companies / organisation are still just in the midst of rolling out Microsoft Office 2010  then you're going to have all sorts of problems trying to introduce use of strict OpenXML as the .docx format and in practice the default transitional OpenXML format will rule for many many years to come. At best, you're going to end up with a messy estate of .docx files some of which are strict format and some of which are transitional format but all are called .docx.

Fortunately if you're using Microsoft Office 2013 this is ok and the more messy the environment, the more incentive there is to all upgrade (including the Operating System). Alas, there are only a limited number of alternative companies providing word processing capable of reading / writing Microsoft Office .docx (transitional format) as it currently stands because of the effort needed to reverse engineer transitional OpenXML (remember strict is the standard).

For example Libre Office does provide the ability to save files as two different .docx formats - one for  "Microsoft Word 2007/2010 XML (.docx)" which is the reverse engineered transitional OpenXML format and one for  "Office Open XML (.docx)" which is the strict OpenXML format that is the ISO standard.

The upshot of this, for many it'll just seem easier to stick with Microsoft, upgrade the OS and Office package, accept a messy estate and in all probability stick with transitional OpenXML as the .docx format. So much for a competitive market and open standards then.

However, there is an alternative - ODF.

So what is ODF? It stands for Open Document Format, it covers a number of office formats (.odt, .ods, .odp) and is an ISO/IEC standard 26300 adopted in 2006. It is supported by multiple technology providers including AbiWord, Adobe Buzzword, OpenOffice, Atlantis Word Processor, Calligra Suite, Corel WordPerfect Office X6, Evince, Google Drive, Gnumeric, IBM Lotus Symphony, Inkscape exports, KOffice, LibreOffice, Microsoft Office 2010 onwards, NeoOffice, Okular, OpenOffice, SoftMaker Office and Zoho Office Suite.

Microsoft also provides support for ODF however as it says "Formatting might be lost when users save and open .odt files". Brilliant.

One of the big advantages of word processing / spreadsheet and presentations systems like Libre Office is they have close to feature parity with Microsoft Office but they also aren't tied to a specific operating system i.e. you can get Libre Office for Ubuntu, MacOSx and Windows.

So why doesn't Microsoft just adopt ODF? In a nutshell, control of a market. Why simply give up a captured market if you don't have to and especially if you can persuade people that the exit costs that your product has created aren't actually an ongoing liability. If you can use those exit costs to persuade people against moving, explain that your product really is 'open' with OpenXML, add in a complex mess of strict and transitional formats then you can hopefully can get people to stay put and just upgrade both Office and the operating system. You get to keep the captured market intact!

Oh, and by the way, those exit costs are significant. Microsoft's own people estimated that adoption by UK Government of ODF as the standard would cost in excess of £500 million. What that means is we're currently locked into an environment which it will cost £500 million to escape from (assuming the figure is correct) but what they fail to mention is that 'liability' is unlikely to decrease over time. It's that 'liability' which keeps us paying for new versions of Office, new versions of OS, application upgrades  and also prevents a truly competitive market for what is fundamentally a common and well defined activity - writing documents etc. 

This is where Government really has to step in because it has power and influence in the market.

This is clearly a mature but captured market which the introduction of an open standard will encourage greater competition, reduce long term liabilities such as exit costs and therefore benefit users. There is no reason why Microsoft can't upgrade previous versions of Office to write strict OpenXML (they've had six years) and change the default .docx to this, however it's probably just not in their interests to do so. The pragmatic choice for Government would therefore seem to be ODF.

If you select ODF then there is also no reason why Microsoft cannot sell office products based upon ODF (which it supports) and if not Microsoft then there's lots of other potential vendors (see list above) including open source solutions who will. Since Portugal and other Governments have already gone down a route of establishing ODF as the standard then it also doesn't make much sense for the UK Gov to create its own island of technology standards. 

But then again, standards are rarely about pragmatism and more about vested interests. Now this is the reason why there will be a fight because you're talking of an established defacto with a captured market potentially being forced into giving up that control and competing in a free market. I expect to see lots of intense lobbying over why OpenXML should be the standard despite the default .docx of Microsoft Office 2010 and 2013 being transitional OpenXML and not the approved ISO open standard strict OpenXML.

At the end of the day this is about word processing, spreadsheets and presentations. We don't need umpteen file formats for this and certainly not two different formats for .docx such as strict and transitional. This will be claimed to be a necessity for reasons of transition but we've had six years and no end in sight to this mess. In reality this is all about prolonging control of a captured market and almost nothing about end user needs.

This is why Governments must act but of course, it'll need you to help counter the well funded  lobbyists that are likely to appear - just like last time.

Before I go, I thought I'd better clear one last thing up. I happen to love Microsoft Office. I use it all the time on MacOSX in particular Excel. However, I'd rather see Microsoft compete on providing me with a better experience / product which was based on ODF (an open standard) in a competitive market than what is happening today. I do understand the temptation to claim something is an open standard when it is not, I do understand the gameplay and how standards can be used to control the market.

I hope, maybe with a change of leadership that Microsoft learns it can compete purely on having better products and not by attempting to create and exploit lock-in for what is a common activity. I fear that this standards debate will just become another example of Microsoft rallying troops into the fray with the usual cries of being "open", concerns over exit costs that ignore long term liabilities and funded "white papers" masquerading as academic work etc.

Hence in the next post, I'll talk about some of the common tactics we're likely to see.