Friday, January 09, 2015

On services ...

When mapping a complex environment, the connections between components (whether activities, practices or data) are interfaces. Those interfaces maybe provided through programmatic means (e.g. APIs), user interfaces (i.e. a webpage) or other means (e.g. physical interfaces). 

However, those interfaces evolve as the component evolves. The interface starts as highly unstable (e.g. constantly changing) due to the chaotic and changing nature of the component. It's novel, it's new, we are operating in the uncharted space of the uncertain. Over time as the component becomes well understood then the interface becomes more stable, even giving an impression of order as the component becomes more of a commodity. For example, a plug and socket are well understood, well defined interfaces which mask the complexity of generating electricity from your average user.

So whenever you map out a complex system or business, you not only have evolving components but the interfaces have different levels of stability (see figure 1).

Figure 1 - Interfaces in a map.

When a component is consumed by others within a more complex systems (e.g. electricity consumed by a computer) then there is a cost associated with any change of the interface or the operation of the component. We experience this (in a trivial sense) when we visit a different country with a different regime of power supply / power standards and we need to take adaptors to cope with the change.

With a large volumes of users consuming an interface in a plethora of complex systems then the total cost of change can be significant. For example, try changing national power supply standards or currency (such as decimilisation in the UK) or the switch to digital TV. Users will tend to collectively resist such a change unless overwhelming reasons are given.

This resistance tends to stabilise the interface and discourage the provider from changing it. This is why your electricity provider can't decide to change the interfaces (e.g. frequency, voltage range) every week and if they attempted to there would be a huge public outcry. This is also why volume operations based services such as cloud services are really only suitable for when the activity has become a commodity and the interface can become relatively stable in a short period of time.

This doesn't mean that you can't provide your brand new "never invented before" concept through an external API to others. However given that the interface is going to change an awful lot in the beginning as we explore what the new concept is, this will limit the volume of users you can support i.e. for some brand new concept then we're talking very few consumers. If you don't believe me, then when you next invent some brand new (never invented by anyone else before) activity then try providing it as a utility service and start changing the interfaces weekly as you discover more about it. The one thing you're not going to build is a volume based service with a large volumes of users. It's simply not ready for it.

Volume based utility services provided through APIs are only suitable for well understood and well defined activities - things like provision of computing infrastructure, CRM, databases, platforms, specific widely used applications etc. These have all had decades to become widespread and well defined enough to be suitable.

Now, when you look at an interface, it simply provides a mechanism of meeting the need for a higher order component (remember these Wardley maps are based upon a chain of needs vs evolution). Hence, from figure 2, the user need is met by component A. However component A needs both component B & C and so forth.

Figure 2 - User Needs and Interfaces

Remember sometimes those interfaces can be provided by APIs, sometimes they are physical interface (e.g. plug in socket), sometimes a user interface etc. Now, in the above diagram, let us consider three user needs - represented by components A, B and D - that we wish to provide as external APIs to others. 

Component D represents a highly evolved commodity-like activity suitable for volume operations where the interface should be relatively stable (an example would be infrastructure such as EC2). These can be provided as utility services and will normally end up over time having multiple providers or ways of implementing.

Component B represents an activity which is in the product phase of competition (i.e. it's still evolving). Hence there will be lots of competition between alternatives on feature differentiation and as a result change. It could be provided through an external API and is more likely to be a very "product" specific rental service rather than generic utility service.  There is usually only one provider for each product type, little duplication of interfaces and no guarantee that your product vendor will end up being the long term utility provider. You can usually see this difference in the language people use, naming specific providers rather than the generic activity. Hence rather than talking about a generic activity such as collaboration, people will often talk exclusively about the specific instance such as Microsoft Sharepoint. This doesn't mean Sharepoint can't be provided as a service, it can and is. However, it represents more of a rental service (often more suited to a subscription basis) as opposed to a utility service.

I know people call both Amazon EC2 & Microsoft Sharepoint both "cloud" services but then that's the awfulness of that dreaded word cloud. Amazon EC2 is far more of a utility service than Microsoft Sharepoint which is still more of a rental service for a product provided over the internet. Collaboration software isn't quite there yet.

You get quite a lot of this "cloudification" of things (for want of a better word). For example, if I take MySQL and run it on EC2 have I created a MySQL Cloud? No, I've stuck a product on a utility service. The process of trying to provide this to a large numbers of users and billing as a utility will certainly end up converting this to a more utility service (databases are evolved enough to be suitable) but whilst the interface to MySQL might remain the same, the system will be very different.

I try to avoid the word "cloud" and instead break down systems into their components. Some are utility whilst others are more products. The two are not the same.

Component A represents an activity which is relatively new (i.e. not widespread and well defined). Though it can be provided through APIs to others, the interface will change very frequently. It really isn't suitable for attempts to provide it on a volume basis through a rental / utility service to others but that won't stop people from trying. This is more suitable for discrete provision to a single or few clients whilst the activity evolves.

I suppose what I want to emphasise here is that the interfaces also evolve as the component evolves. That's a fairly obvious point but one to emphasise. Also when we talk about a services based organisation, then in the technology sense we're mainly talking about building an organisation where many of these interfaces are provided through APIs. However, just because you provide APIs doesn't mean all APIs are the same - some are inherently more stable than others due to how evolved the activity is. You need to factor this in.

The best guidance to building a services based organisation was apparently written in a memo by Jeff Bezos in 2002. I've not seen better since, so I thought I'd copy the spirit of that memo here.

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn't do this will be fired.

This was 13 years ago. I just thought I'd emphasise that point because I hear people talking about building a services organisation (e.g. use of micro services) as though this is some radically new idea. I'm not against this, it's a good idea but just be aware - it's not new and this is more about keeping up with the pack rather than gaining an advantage.