Building an Interoperability Layer ...

Over the last few years, the question of portability in the cloud computing world, has been growing steadily louder. I tackled this particular issue at OSCON in 2007. It is good to hear that Tim is advocating the building of an "interoperability layer".

With one exception, the only way that we can achieve such portability in the cloud (whether at the application, framework or hardware end of the stack) is to have open sourced standards, where the standard is a fully operating open sourced piece of software.

The Open SDK for Google's AppEngine is a fledgling example of this concept. The Open SDK is the standard and AppEngine merely one implementation of it. Within a few days of its release, it had been re-implemented elsewhere.

Open source is an essential part of portability. Though I'm supportive of the general aims of groups like DataPortability, I'm opposed to their approach. Focusing on "open standards" diminishes the value of "open source" and provides a simply marketing tool for describing a computing cloud as "open". This approach may well lead to more not less lock-in. Open standards are necessary for portability but they are not sufficient for it.

Now an open sourced standard doesn't mean that what is built upon it has to be open sourced. However as we move towards a more service based economy in IT, this is likely to become more common. This shouldn't be seen as somehow diminishing the software industry, but instead allowing for the creation of entire new industries based upon an ecosystem of competing providers. Such competitive utility computing markets will require monitoring and compliance services and allow for exchanges and brokerages. Innovation within those providers will occur at the operational rather than the product level, with competition based upon price vs QoS (quality of service) rather than feature set. This will allow for more efficient use of resources, a more effective balancing of supply and demand, and further innovation within society.

This move towards commoditisation of ubiquitous and defined services isn't a negative thing. It's going to create a wealth of opportunity, as long as we avoid the interoperability disaster of different clouds all built on different "secret sauces" with no open source standards and hence no functioning marketplace.

Few companies are ready to enter this world which competes on service rather than product. Few are ready to encourage competitors and to create an ecosystem.

This is an opportunity that is ripe for harvesting - "Carpe diem"

Ready, Steady, .... Dither ....

I first came across BungeeLabs at Web 2.0 Expo (April 2007), when I was presenting a re-run of my earlier talks on commoditisation, competitive utility computing markets and SaaS.

After handing over several T-Shirts of pre-shaved Yaks, I had a long discussion with them over why I thought their decision not to open source was a mistake and my company's plan to release an open source federated grid at OSCON'07.

So I was interested to read recently that BungeeLabs have "announced federated hosting to expand adoption of Platform-as-a-Service (PaaS) among enterprises" which will enable users to "self-designate the hosting location"; that's good news. They've also got round to looking at that open source question again.

BungeeLabs have a chance of creating a small piece of an enormous pie (rather than all of a small one). This opportunity depends upon them moving quickly to open source, encouraging other providers to setup as competitors and hence creating an ecosystem with users choosing and swapping between hosting providers.

If they get such an ecosystem up and running, then once it reaches a certain size of providers and consumers, it will become self sustaining. Huge new opportunities will arise in the management and maintenance of this marketplace. The utility hosting aspect is small fry when compared to the potential exchange, brokerage and futures opportunity.

Technically this is all fairly easy, however, it does take a significant force of will and conviction to make such a move. They have left it late to make such a move and a big player may enter this space before they can get started, I wish them good luck.

Fortune favours the bold, opportunity is fleeting and don't dither!

What's the future ...

The golden era of the internet has been built on a culture of open source. This culture has removed barriers to adoption of technology and heralded in an age of unprecedented innovation in society.

As technology has become ubiquitous, we have tended to view it as more of a commodity, a common and re-usable component or service.

We are now seeing the growth of a service based economy (SaaS) and more service oriented approaches to development. These changes will encourage further innovation and open the door to competitive utility computing markets, more effective and efficient usage of resources, portability between providers and competition based upon service rather than product.

This is a living nightmare for those who have seen their competitive advantage as being their technology. Many are simply not prepared for competition based upon service.

As with Google, Microsoft has now entered the field with the launch of their Mesh. I'll note :-

• there is no open SDK (yet) with the Mesh, the backbone of the technology is 100% Microsoft.
• the Mesh is described as a platform for the web equivalent to how windows was for the PC.
• it provides many opportunities for collaborative business benefits (a highly attractive feature for adoption).
• Microsoft has a strong channel of ISVs and is looking for ways to support this.
• it describes a future hybrid model between in-house and cloud infrastructure (the P2P aspects seem to go beyond file sharing).
• it can mesh multiple devices and the cloud, and presumably allow ISVs to become resource providers (hence creating a marketplace within the Mesh).
• they are seeding this Mesh with their own significant infrastructure.

What we don't know is whether Microsoft will be adopting an open approach, encouraging an ecosystem of providers and competing solely on service.

Interesting times lie ahead ...

More *OA with your *aaS?

Hot on the heels of * as a Service is the new and exciting name game of * Oriented Architecture.

This time, the game is a bit more tricky as you need to redefine old terms in order to allow new terms to fit in. So Service Oriented Architecture (SOA) is no longer about Service Oriented Architecture but instead about the use of acronyms like SOAP & BPEL, excluding other acronyms like REST & RIA

Lost? You should be.

This new package holiday of confusion even comes with colourful jargon like:-
"On a cruise ship, the WOA folks would be the recreation directors, chefs, entertainers, and cruise directors making it a fantastic journey. The SOA folks are working down in the hot, noisy engine room, making the ship go forward.".

Wow, let's hope you end up with the fun loving recreation people rather than those smelly old engineers (see figure 1 & 2).

Figure 1 - A typical recreation loving WOA Architect.
(click on image for larger size)

Figure 2 - A typical hidden-in-the-engine-room SOA Architect.
(click on image for larger size)

(Scenes of Morlocks and Eloi from the The Time Machine.)

For anyone requiring a bit of sanity, let me simplify it all for you.

Software as a Service is a delivery model for providing software. In today's world, the software stack is undergoing commoditisation from a product based economy (i.e. Software as a Product) to a service one (i.e. Software as a Service). This can be at any level of the stack including but not limited to application, data, framework and hardware, hence all the derivative *aaS's.

Service Oriented Architecture refers to the use and construction of services (or common, higher order components) in the construction of new services or products. The important things to remember are architecture (a deliberate or planned approach) and oriented (organised around) and services (using services). In other words, a deliberate or planned approach organised around using services.

These services might be internal or external or they might be SOAP based XML or RESTful web services using a REST architectural approach. The other *OA variants are simply subsets of SOA.

This is where the pantomine usually starts; "oh no they're not, oh yes they are" and so on. To avoid this argument, simply raise your hand if you are using an ROA or WOA. Now drop your hand if it uses services of some form or another. If you've still got your hand up, you're a *OA variant that is not a subset of SOA and I'd like to hear from you.

As for the hip and cool cruise directors and the engineers hidden in the depths of the dungeons, I prefer the somewhat less controversial terminology of Pioneers (Eloi) and Town Planners (Morlocks) when describing the people who deal with different types of organisational activities (see figure 3).

Pioneers explore the unknown and boldly go whilst Town Planners build cities. Both use common services if they've got any sense.

Figure 3 - Less controversial roles.
(click on image for larger size)

Preparing for XTech ...

XTech is in two weeks time, so I've been busy preparing my talk.

As part of the process, I've provided a preliminary paper on the topic I'll be covering.

What's your perfect job?

Recently, I was asked the question: "if you went back to working for a company, what would be your ideal or perfect job?"

Obviously this will change with time, but today it would be a role which played to my strengths and interests both in management, strategy, innovation and commoditisation. So I wrote down my perfect job and discovered that it doesn't exist ... yet.

To make matters more complicated, the two roles it would work alongside - Chief Innovation Officer and Chief Planning Officer - are rare themselves.

Fortunately for me, the life of a freelancer is one that you create yourself. It's less of a job and more of a lifestyle. If you want to find out more about Going Solo, then I'd recommend attending Stephanie Booth's conference.

Still, at least it's interesting to know what it would be. What's your ideal or perfect job?

U.S. vs us ...

In some European technical circles, the U.K. is sometimes considered not relevant to Europe and just a poor cousin of America. This poor cousin attitude is often reinforced by the claims from numerous bandwagon hopping analysts that if you want to make it big then you need to go to Silicon Valley.

Contrary to such blatant stereotypes, the U.K. (and Europe as a whole) has a thriving technical and entrepreneurial community with companies such as Dopplr, Last.FM and Xing to mention but a few.

Despite this, the government body UK Trade & Investment, has provided financial support to web mission 2008 for "20 UK Web 2.0 companies [to] travel to San Francisco to explore new opportunities for growth with key people in Silicon Valley".

As Ryan points out, whilst the intention is good, the whole web mission basically declares that "we don’t have what it takes over here”. The reality is we do, so why not invest in a U.K. based activity and promote the start-up culture here?

Same old, same old ...

John Maynard Keynes once observed that;

“Capitalism is the astounding belief that the most wickedest of men will do the most wickedest of things for the greatest good of everyone.”

I happen to agree. The works of J.M. Keynes, Adam Smith and J.K. Galbraith are all founded on the belief that the market is merely a tool for society and not a replacement for it. Unfortunately in the last few decades we seem to have lost that message.

Our Government is determined to dispose of the 10 percent tax rate (a key part of a progressive tax system), despite fervent opposition and evidence of the harm it will cause to the common people. At the same time that it is penalising the common people (or society), our Government is quite happy to bail out the reckless gambling of the banking community (and the market). It's enough to send Sidney Webb spinning in his grave.

Though this wicked change will harm the poor, I want to take a moment to remember an upcoming anniversary of another wicked act, our treatment of the poisoned people of Camelford.

If you are unaware of this incident, let me remind you. In July 1988, the then government was rapidly privatising a number of industries for reasons of political dogma. During this time, 20 tonnes of aluminium sulphate was dumped into the wrong water treatment tank at Lowermore.

The fresh drinking water for over 20,000 homes in Camelford suddenly turned into a poisonous cocktail. Despite livestock dying and peoples' hair turning blue, the soon to be privatised water authority insisted for several days that everything was safe.

Over the following few years, a government appointed health group investigated the incident and concluded that there was no long term health affects. The medical conditions being suffered by the residents of Camelford, such as memory loss, hair loss, joint pains and general exhaustion were not due to poisoning but were instead caused by anxiety and the mass hysteria resulting from articles in the popular press. This conclusion was gibberish and a convenient whitewash during a time of intense privatisation.

The residents of Camelford have never been properly compensated, their treatment has been appalling and no-one really knows what the long term health effects are.

Had the Government at that time cared more about the common people (or society) as opposed to its privatisation programme (and the market) I wonder whether the outcome would have been any different.

The answer is obvious, but it does also question what has actually changed since then.

It's time things did.

More from less ...

In 1970, at its Palo Alto Research Centre, Xerox had arguably assembled one of the most creative groups of computer scientists the world has ever seen. This team created a computer system that most would consider to be more than a decade ahead of the rest of the industry. Email, laser printing, graphical user interface and a mouse. No-one else came close. Over 1500 of these computers were in active use within Xerox

So why doesn't Xerox own the computing industry today? As Steve Jobs famously described, they effectively “grabbed defeat from the greatest victory in the computer industry".

Whilst PARC played its role and brought the future to Xerox, the senior management of Xerox failed spectacularly to take advantage. The more I research into this subject, the more I discover that this is not an isolated incident. This phenomenon appears to be widespread. Whilst failure is an intrinsic part of the process of innovation, and that includes failing to act or implement an idea or take advantage of it, there are also many often unnecessary obstacles in its way.

Despite senior executives calling for more innovation, in most organisations it doesn’t seem to get an easy ride. Reasons for this include a lack of experience with radical innovation projects at senior levels, a growing mismatch between R&D productivity and cost, and a disparity between how long innovation takes and the immediate demands for ROIs. Added to this are common excuses used to stonewall innovation, from the ever faithful tyranny of current strategy (“it's not core”), to arbitrary financial hurdles (“it’s not worth our time”). Even if your innovation manages to navigate this minefield, it often receives the coup de grace from internal politics or simple fear.

As Machiavelli once said;

"the reformer has enemies in all those who profit by the old order".

Almost everyone who has ever tried to do anything innovative in a large organisation has at some point collided with the organisation as "super tanker" metaphor; we might have no idea what lies ahead but we certainly can’t (or more likely won’t) change course quickly.

I've become increasingly convinced that what CEOs should be crying out for is not more innovation but fewer self-imposed obstacles.

Now that is something they can fix.

Here comes the farmer ...

In 2005, I started an ambitious project to build a utility based development environment, which was successfully launched in early 2006.

Back in 2006, I described the service as a "platform for developing SaaS like apps in the future". This service was identical in concept to Google's recently released AppEngine, except that we used JavaScript and our plan included the open sourcing of the entire stack and not just an SDK.

Within our service both code and data were stored as portable objects. The open sourcing of the product was planned to provide alternative places that you could port to. I nicknamed this concept FaaS or (Framework as a Service), though Robert did warn me that we would just end up with a lot of aaS.

He was right.

For anyone confused by the abundance of terms like DaaS, PaaS, IaaS, HaaS and so on, the reality of what is happening is very simple.

Any X as a Service simply refers to the commoditisation of the computing stack from a product based economy to a service one. End of story.

A key part of this transition is that we move to a service economy that competes on service rather than product. Such a transition has enormous benefits for society in terms of innovation (I'll talk more about that later in the year at Web 2.0 Strategies). However this is a difficult transition for those who have always seen their product as the source of advantage.

In a service economy, SLAs are not as important as portability between providers. Without such portability you will remain still stuck in a product based economy, albeit one that you can rent over the wire. The companies who accept that service is the key to competitive advantage in a service economy will have no problem embracing this; those who believe their technology is their "secret source" will always have problems adapting.

Where for example, two years since its release, is the alternative provider for the Amazon patented S3 environment?

Like it or not, we are moving towards a world where ubiquitous IT activities will be provided by utility computing markets. Providers will openly compete based upon price vs QoS (quality of service) and services will be defined by open sourced standards.

A lack of open source was one of the things which concerned me about BungeeLabs offering. Now I gather from James Urquhart's blog that Intuit has entered the fray with its own platform which builds upon QuickBase's database and workflow applications. Whilst Intuit are making positive noises towards portability, the way to actually achieve this is through multiple providers and easy switching between them.

In my view, if Intuit want to have any form of real lasting impact in the shift towards a service oriented economy and avoid being another "also ran", they should open source their entire offering immediately and take an approach of competing on service.

Words are grand, but as per Aesop's fable of the farmer and the crane ...

... actions speak louder.

P.S. If you're thinking why would an established company even consider such a route, I'd recommend reading the section on, "Why Great Companies Can Fail" in Christensen's book, "The Innovators Dilemma".

P.P.S. If your product doesn't become the open sourced standard in a service oriented economy, someone else's will.

One Social, one not ...

This morning I caught up with Stefan whose project Soocial is really starting to take off. This is fantastic news, they're a wonderful bunch of people and hopefully they'll become the internet superstars they deserve to be.

That was the social or good news.

The anti-social or bad news came in the form of an article by the Washington Post on the use of waterboarding in team building exercises. Apparently the assembled sales team were told:-
"You saw how hard Chad fought for air right there. I want you to go back inside and fight that hard to make sales".

What planet do these people come from? I wonder if Chad's medical insurance covers acts of management insanity?

What next?
"You saw how hard Chad fought to avoid being shot. I want you to go back inside and fight that hard to make sales."

What happens if Chad doesn't quite make it?

Figure 1 - The future of team building.
(click on image for larger size)

Apparently the general counsel for the firm said that they weren't the mean water boarding company that people think they are. It takes a lot to make me swear, but well who the fuck are they then? I'd hate to meet a really mean company if waterboarding is okay for the nice ones.

He also asked whether this would even be an issue "if it weren't for Guantanamo Bay". Let me think about that one - try, yes it is an issue if you go waterboarding your staff in order to make them work harder.

The best line however is when he talks about how this incident is "going to hurt our image". What about your staff?

Save Chad now before it's too late.
Thanks to molumen and Gerald_G for the public domain clipart that I've used and Marilyn Pratt for spotting this.

FLOSS saves industry billions ...

Apparently, FLOSS (Free/Libre and Open Source Software) has saved industry over $60 Billion according to this announcement by Standish Group International.

I'm guessing this is just cost savings in license fees and excludes all the amazing new services and innovations created by people in the FLOSS world. I'm guessing, because unlike all the good work which goes into FLOSS, the Standish Group International's report costs $1,000. I hope they weren't commissioned to produce this report.

If they were, then please, next time, commission an open source group or an academic group or an analyst group like RedMonk so it can be published freely.

Of course, in the future, such data will be a freely available commodity. As new mechanisms for data capture and interaction grow (through social networks to web services to spimes) it will become increasingly important for a company to become the canonical source of information about itself. Companies are likely to make a unprecedented shift towards openness and transparency. This will be good for innovation in society but it will obviously upset those who are not willing to adapt or who make their living by selling aggregated company information back to companies.

N.B. That was not a "prediction", it is already happening.

According to the report, the $60 billion in savings only accounts for 6% of the annual spend; so there is plenty more scope for improvement with a bit more investment and corporate collaboration with open source groups.

Excellent news.

Original report picked up from a tweet by Dion Hinchcliffe

Commoditisation & Commodification

I thought I'd just re-iterate the distinction between the above terms that was first identified by Douglas Rushkoff:-

• Commodification (mid to late 1970s, Word) is used to describe the process by which something which does not have an economic value is assigned a value and hence how market values can replace other social values. It describes a modification of relationships, formerly untainted by commerce, into commercial relationships.

• Commoditisation (early to mid 1990s, Neologism) is the process by which goods that have economic value and are distinguishable in terms of attributes (uniqueness or brand) end up becoming simple commodities in the eyes of the market or consumers. It is the movement of a market from differentiated to undifferentiated price competition, from monopolistic to perfect competition.

These two processes don't just occur to physical things. For example, where knowledge or skill can be codified, as in written down in a manual, it will most likely become traded. This act commodifies such knowledge. Of course, the cost of reproduction of knowledge in a digital age is almost zero, so such knowledge tends to spread. It therefore becomes commoditised, even to the point of being given away freely in wikipedia.

Commoditisation applies to physical capital, human capital and even social capital.

Where do you think reputation based networks are going to lead if not towards the codification, commodification and subsequent commoditisation of social capital. Did anyone really believe that commoditisation of content would stop at news and somehow ignore film, music and other content based industries.

I more than understand the concerns of the "old guard", but I'm afraid that's change for you. The wandering minstrels, town criers and hot metal machinists of the past got used to it, and so will those who face the cold wind of change today.

Commoditisation effects all industries that are intensive in the form of capital that is being commoditised.

Of course, no-one likes this, hence the brouhaha we have in the IT industry about utility computing clouds. This is simply our industry following a well trodden path, as was predicted back in the 1970s. I say predicted because in reality it's just the same old pattern of commoditisation applied again. I covered many of these themes in my Future of Web Apps talk in 2007.

The three stages of expertise ...

are "I know nothing", Hazard and "I know nothing" .

Figure 1 - The three stages of expertise.
(click on image for larger size)

P.S. Just in case, the above is meant to be a joke and a social commentary on the hubris of self declared experts. It's based upon precisely nothing. Just because someone draws a graph, doesn't make it real.

What's behind that door?

I'm glad to see that a petition for using Open Document Formant (ODF) in any National Archives has started. As a reader at the British Library I personally think we should be adopting open standards for document formats.

But wait, isn't OOXML (open office XML) an ISO standard!

ISO certainly approved OOXML, now renamed OXML (minus the "office" bit) to be a worldwide standard. However, it is questionable whether this simply means that ISO approval can be bought by stuffing its voting committees with sympathetic members and whether ISO's reputation is in tatters.

Some groups have even started to protest this standard which it is claimed forces "Microsoft customers to keep buying the American software giant's programs".

Can ISO recover its battered image? Well, OXML is now in the hands of ISO, which has set up a harmonization group to deal with it.

Who is in charge of this group? They've selected Dr Alex Brown who described OXML as a "baroque cliffside castle replete with toppling towers, secret passages and ghosts".

Let's hope those secret passages don't lead us to the sort of proprietary lock-in nightmare described by Mr. Edwards

Why open matters ....

XTECH is only a few weeks away. I'll be giving the opening keynote on Wednesday during which I'll cover why "open" matters, from innovation to commoditisation.

I'll be exploring a number of themes from:-

• How open source is accelerating the commoditisation of IT.
• Why open source is essential for competitive utility markets.
• Why open standards are necessary but not sufficient for data portability.
• How the spread of the open meme is not only accelerating innovation but undermining traditional forms of business.
• The growing conflict between organisation and innovation.

So do say "Hi!".

I won't be speaking at OSCON this year, due to a miscommunication and then me agreeing to do some other work instead, then being told I was invited to speak, and then me not being able to afford to ... yada yada yada ... long story, very dull.

Upshot is that I won't be doing any U.S. gigs. Obviously the "hummer" joke didn't go down that well.

I do keep a list of upcoming talks and videos of previous talks. If you want me to speak at your event on any of these subjects, I'd be happy to hear from you.

Lock-in? What lock-in?

And so a brave new world of utility computing markets makes another step with AppDrop, a "proof-of-concept to show that Google's App Engine platform can be ported to other hosts"

Read more about it at Chris Anderson's blog. There is also a write-up by Andy Baio.

N.B. I said starts because this isn't true portability or an example of a competitive market. This is just another step heading in the right direction.

Google Lock-in?

Just noted that Tim asked the question whether GoogleAppEngine was a lock-in strategy?

I've made a comment on the radar, but I thought I'd leave a copy here for reference.

=====

Hi Tim,

I feel like I'm pointing out the obvious and just repeating what I talked about at OSCON. Then I realise you already know all of this and are just getting the conversation going.

In my view, the real innovation in GoogleAppEngine (GAE) is the open SDK and the possibility that open SDK compliant environments can be re-implemented elsewhere (including on Amazon’s web services). The difference between Google and Amazon is that Google has taken the approach of creating an open sourced standard and in effect encouraging competition. Amazon has always been about their “secret sauce”.

Whilst GAE is at the level of a framework and Amazon is at the level of infrastructure, the approach of an open sourced standard, if adopted, is likely to spread up and down the stack whether this is at SaaS, PaaS, HaaS or whatever other aaS people choose.

Jaws 3.0

Forget sharks with friggin lasers (Jaws 2.0), think sharks with friggin lightning bolts.

Sparta ....

I wasn't going to post any more on this subject, that didn't last!

I believe that not open sourcing the entire GoogleAppEngine stack is a tactical gamble because there exists the danger of a proprietary cloud. This is an interoperability disaster that could threaten the entire freedoms which have been established on the internet.

Is it going to happen? It can only happen if:-

• companies and people are short-sighted and gullible enough to fall for it.
• a big enough player releases a proprietary software stack which meshes together multiple data centres and even devices into a single P2P cloud controlled by their software stack.

Well, Microsoft is releasing some new cloud service. Fingers crossed that it's not heading down the path to some proprietary P2P cloud. Let's hope they will be releasing an open SDK equivalent or even better, open sourcing their stack.

Random connections ...

Every now and then I read a combination of random articles that in total make me shiver. Today's chilling trio are:-

• Japanese firms are "creating the world's first ratings agency looking at data security" in other companies.
• Autonomy releases a "Policeman Inside Your Computer And Inside Your Corporate Blog"
• an American consultancy is accused of "bringing 'union-busting' tactics to Britain"

Now this reads like some sort of 1984 dystopian nightmare. I've noted that the American consultancy involved, the Burke Group, believes that "human resources issues are a collaborative process of partnering".

Well I agree. Collaboration is often an important factor in success, however it is also worth remembering that people innovate and not companies.

Without companies, people are still people but without people, companies are nothing.

So what should you do if you're faced with a Burke & Autonomy & Rating nightmare? The most powerful weapons we have are openness and collaboration. Share your own ideas with the rest of the world and if you want to make a change in your company, form a union.

Nothing worthwhile comes easy, you need to fight for it. Fortunately those good people at Retroshare have released a new version of their system for the private broadcasting of messages.

GAE roundup

For me, the real innovation in GoogleAppEngine (GAE) is the provision of the Open SDK which is basically an open sourced standard. This open SDK can be re-implemented on other environments, even Amazon's, and hence is a first real step towards portability in the world of SaaS, PaaS, FaaS or whatever other aaS you desire.

Anyway, whilst it is an interesting distraction from my current work on innovation, it is already something which I've talked about for many years. I've provided links to videos for my OSCON & Web 2.0 Expo talks for those who are interested in more.

Video of OSCON talk - Commoditisation of IT (approx 40 minutes)

Video of Web 2.0 EXPO talk - Short on Storage, Long on Cycles
(approx 40 minutes)

I've read a number posts about GAE over the last few days, most of them seem to focus on the details but I've picked a few places that I consider are really good:

• Jesse Robbins raises some interesting questions about utility computing.
• Tom Gleeson provides an excellent analysis of Python as the new VBA
• Niall Kennedy provides a good analysis of what GAE is from a developer's perspective.
• Dion Hinchcliffe provides a good comparison between Amazon and GAE
• Doug Cutting examines open sourced implementations of cloud computing standards.
• James Urquhart provides a good analysis on what GAE isn't.

Amazon + Google

Earlier I posted about Amazon vs Google. Which is a classic case of me not seeing the obvious.

Anyone planning on re-implementing the Open SDK of Google AppEngine onto Amazon's web services (EC2 / S3 etc) in order to give us multiple providers and a second step towards portability?

I'm giving a keynote at XTECH on why open matters. If anyone is planning to create an open SDK equivalent on Amazon - let me know.

Columbus, I'd like a detailed plan before you go ...

When it come to managing an activity, there are three wise sayings that I always keep in mind. The first and second are:-

• proper planning prevents poor performance.
• an imperfect plan executed today is a better than a perfect plan executed tomorrow.

The former simply tells you to make enough of a plan. Obviously for a commodity-like activity, which is well defined and ubiquitous, such a plan can (and should) be quite detailed. For a genuine innovation that plan becomes more of a fuzzy haze of intentions; it's an undefined activity which will change.

The latter wise saying simply advises against missing an opportunity because you've spent all your time planning. Opportunity, as in a genuine competitive advantage, is often fleeting.

Now what is true of planning, is also true of mapping. When mapping a business, it is worth remembering that some are commodity-like activities that can (and should) be mapped in detail, whilst other processes are innovations which tend to be more of a shifting smudge.

For me, the purpose of mapping out activities, beyond process re-engineering and the normal operations, is to enable the organisation to build a framework of services on which future innovations can be built. However, embarking on a Service Oriented Architecture (SOA) without some form of map is like saying "we're going to build common services but we don't know what we do" (see wise saying No. 1).

However you can't wait until you finished a perfect map before you start, because within any commercial ecosystem activities constantly change (see wise saying No. 2) and to be honest, you've got those smudges. So you'll have to do some exploration.

The reason why most organisations adopt a service oriented approach to design is because componentisation of business activities as services is a powerful enabler of innovation (part of the theory of hierarchy). Columbus needed the resources, organisation, infrastructure and might of Venice before he could embark on his exploration of the unknown. Though certain activities (such a ship building) can be planned in detail, the discovery of a new territory cannot. If Columbus had been required to create a detailed map before he left, then he'd never have left.

In business, the discovery of a new territory, such as a new product or service to sell, is essential for the survival of company. This brings me onto the third wise saying:-

• innovation requires organisation to survive and vice versa.

Open source as an innovation strategy

Innovations are the first-ish attempt to put an idea into practice. They require deviation from the normal way of doing something and hence someone to go against the current school of thinking. Sometimes they succeed but mostly they fail.

Every now and then, I come across an idea that seems so wrong that it leaves me pondering whether I am missing the point. One such idea is that of open source as an innovation strategy. To explain why I think this is so wrong, let's first explore what we know about open source and innovation:-

1. Open source allows for the rapid diffusion of an innovation by removing barriers to adoption of any particularly technology.
2. The diffusion of an innovation leads to its commoditisation and the emergence of standard services or components.
3. Components and standard services allow for more rapid adaptation and evolution (Theory of hierarchy)

So we can logically argue that open source accelerates commoditisation of a technology and hence accelerates the process of innovation in a society by providing stable components or services on which new innovations can be built. You can call this the "building on the shoulders of giants" hypothesis.

We also know that:

1. As any innovation proves useful, competitors will attempt to copy it.
2. The competitive advantage of any innovation depends upon its scarcity, hence as it spreads its strategic value diminishes.
3. There is a cost associated with maintaining a home grown system vs a growing industry standard.

So, open source can be used as a tactical play. For example, one might exploit a technological distinction until competitors start to use equivalent systems and then subsequently open source one's technology in order to become the industry standard and avoid the dreaded "cost of migration". This is a "maximising competitive advantage through open source" tactic.

We also know that:-

1. Open source allows for the rapid diffusion of an innovation by removing barriers to adoption of any particularly technology.
2. The diffusion of an innovation leads to its commoditisation and the emergence of standard services or components.

Hence you can use open source as a disruptive tactic to create a standard in an already emerging market.

We also know that:-

1. The competitive advantage of any innovation depends upon its scarcity, hence as it spreads its strategic value diminishes.
2. Open source allows for the rapid diffusion of an innovation by removing barriers to adoption of any particularly technology.
3. The diffusion of an innovation leads to its commoditisation and the emergence of standard services or components.
4. A competitive market for a service oriented technology is only achievable through multiple providers complying with emergent standards and ensuring that there is portability between providers.
5. Open sourced standards provide a means of achieving this without providers handing over strategic control of their businesses to a third party.

Open source is an essential element for any future competitive utility computing world.

Of course, none of this is new. Open source has long been used as a hunting ground for exceptional technology talent, a means of collaborating with other groups and as a powerful tactical weapon.

Along with public speaking on these matters, I've purposefully used such tactics in the past and I know that most of my compatriots do as well. These tactics are common knowledge for most CIO / COO / CEO in small European technology companies exposed to the open source world, and it has been so for many years.

Open source :-

• is a powerful tactical weapon.
• a great leveler of the inequality between the distribution of opportunity and the distribution of ability.
• promotes innovation on a societal basis.
• can create common components and services.
• is an essential element of a competitive utility computing market.

It is therefore likely that more and more companies will learn how to truly use open source; not just as technology but also in the tactical battlefield with others.

Open source is more likely to become the norm.

However, innovation is deviation from the normal way of doing something. It is the first-ish attempt to do something new. A strategy for innovation needs to be based upon influencing such behaviour, it is not about whether some practice or thing is itself new or not.

At the level of society, open source is an accelerator for innovation, and hence the conflict with patents (a system which in many industries has already served its purpose and needs to be removed). At the level of a company, open source is no more of a strategy for company innovation than closed source. It is instead a powerful tactical weapon. Whilst I would agree with Mike Milinkovich that companies should band together to develop common open-source solutions to standard or commodity-like activities, as I've said before, knowing the stage of an activities lifecycle is paramount to tactics and management. I've covered this in more detail in the following video.

Managing a complex world (45 mins)

Open source is a critical business issue, a powerful and positive force for innovation in society and an essential tactical weapon for any corporate. Management however is not about simple linear methods, and if you're thinking that your new innovation strategy should be "open source", then I'm afraid you're missing the point.

I'll talk more about this at XTech.

P.S. This is not the same as open innovation markets, which are a completely different discussion.

More on Google App Engine ...

It's not about whether it's ready for business today, or whether it has this language or that language, or even about lock in issues of SaaS, something which I've talked extensively about.

The innovation in GAE is in the provision of an open SDK and how open sourced standards can provide portability. Whilst I understand the concerns being raised by pundits on various forms of PaaS (platform as a service), I think a significant point is being missed.

People seem to be so immersed in the detail of GAE, that they're missing the bigger picture, even though it's jumping up and down shouting "look at me, look at me!" with a sign saying "big picture" wrapped around its neck.

Either that or I've gone mad.

What is platform as a service?

I was asked today, what is "Platform as a Service" (Paas). Apparently there has been a lot of talk about this subject recently.

After a quick bit of digging, I discovered that PaaS seems to be a term coined by Salesforce, to describe a platform for building software as a service.

It's what I used to call framework as a service. I think Nick is right, there really are too many aaS's. I've tried to express this in Venn form in the following diagram.

Figure 1 - What is SaaS?
(click on image for larger size)

In reality, these various forms of "as a Service" simply represent different aspects of the commoditisation of the software stack from a product based economy to a service one (see figure 2). Don't get bogged down in the different acronyms being used; the underlying concepts behind this change are the important thing to understand.

Figure 2 - Understanding aaS
(click on image for larger size)

If you want to know more about this subject, then my presentation from OSCON last year (though it is fairly old) provides a basic introduction.

Video of OSCON talk - Commoditisation of IT (approx 15 minutes)

Run Rabbit, Run Rabbit, Run, Run, Run ....

Four years ago at EuroFoo 2004, I talked about two subjects. One was fabrication technologies and 3D printing. The other topic covered why much of IT was little more than a cost of doing business. Both subjects were in fact about commoditisation.

For me this has been an ongoing discussion for the last decade, a fair chunk of which has been with a good friend of mine, James Duncan.

In 2005, we started to build a service oriented business known as libapi, consisting of utility computing web services. As this concept developed it changed (a normal activity of any innovation) and became a utility computing cloud with an application framework known as Zimki. It's what we called a Framework as a Service (FaaS), just write your application and then release. 

All that Yak shaving stuff from scaling to configuration to infrastructure was taken care of for you. Zimki even ran on our own private HaaS (Hardware as a Service) known as Borg, something we had started building in 2003. But as a user of Zimki you just worried about code and data, everything else was taken care of, all the yaks came pre-shaved and the shaving was hidden away.

At the heart of the system was the concept of a portable environment. There were many amazing moments along this journey, including :-

• Tom writing and releasing a new form of wiki from scratch to live on web in under 25 mins (2006).
• James demonstrating true portability of a running application between multiple Zimki installations (2007).

The important part of this tale is that we showed that building an ecosystem of utility computing providers, with simple portability of applications and code between those providers, was technically feasible. Such an ecosystem is what I call a competitive utility computing market.

Whilst, "open" standards (as in APIs & data formats) are necessary to achieve portability, they are not sufficient to achieve this. You need to have not only common APIs, but multiple providers and assurance that portability is possible i.e. semantic interoperability. Open source provides the fastest means of operationally achieving this and gaining adoption. The provision of an open sourced environment at any level of the stack (from SaaS to FaaS to HaaS, or any of the other aaS's) is what I refer to as an open sourced standard.

With an open sourced standard you can become a provider by either installing the standard (it is by definition a fully working stand alone environment) or by creating your own version which complies with all the primitives of the standard.

The standard provides a means of ensuring compliance between environments through monitoring.

The open SDK of GoogleAppEngine (GAE) is the potential beginning of an open sourced standard, assuming someone turns it into a fully functioning and scaleable open source system. Any application built to work in the open SDK of GAE will work on any provider's environment which complies with the open SDK of GAE. 

This is not about this language (python) or that (ruby) or whether it's ready for the enterprise or whether it's EC2. In my view, it should all be about creating an open sourced standard and then Google seeding an ecosystem with its own implementation of that standard. Now obviously this is not enough to ensure portability as you still need monitoring and assurance but it's an essential element for any utility computing market which is going to be a free market i.e. it is not going to be based on some proprietary technology and hence a nightmare of control and constraint. 

Hold on, isn't Google's implementation of the open SDK proprietary? Yes, but the open SDK is, as it says, open. You should look at this as though the open SDK is the beginning of an emerging standard and GAE is simply Google's implementation of this. Personally, Google should have started with an open source GAE because they run the risk of someone building an open source play against them. 

This is just the beginning of what I discussed at OSCON in 2007. Ok, there are a lot of ifs and buts, however it certainly has exciting potential. If other providers create environments that are compliant to this open SDK, then we will have the beginnings of such a market. In my book, these changes have the potential to disrupt the entire enterprise IT and hosting market because they're ill prepared for the inevitable.

Like it or not, most ISPs are facing a bleak future at the hands of either huge new service providers or possible lock-in to a proprietary mesh. The only way that such ISPs can fight against this is to create a marketplace based upon open sourced standards. As a marketplace they can fight; as lone providers, their days are numbered.

Both barrels of change - innovation & adaptation - are pointing firmly in their direction. They need to be seriously thinking about creating open sourced standards, a monitoring body and a competitive marketplace. The time to do so is rapidly approaching; you can't hide forever and creative destruction has you firmly in its sights.

P.S. If you want to keep up with the latest developments in this field, I personally read James Urquhart's and Rich Miller's blog.

P.P.S. Yep, I know I'm repeating again a lot of stuff I talked about last year, but I thought it was timely.

-- Update 19th July 2013

Gosh, this is over five years old. Since that time -

• Google App Engine now has AppScale and a development effort with RedHat but is also has serious competition with Cloud Foundry which is playing a fully open source game. Never quite understood why they didn't start the game with an open source GAE, way back when.

• All the terms are wrong. HaaS became IaaS, FaaS became PaaS etc. Don't ask why, I never understood.

• I finally gave up using the term 'innovation' for the novel and new because that word is used for pretty much everything these days. "The bus went a different route today - it's an innovation!" etc.

Sicko ...

Watching the Michael Moore film on health care in the U.S.

Thank god, I live in a more civilised country. It's not perfect but generally it's very good.

I know we have a lunatic economic fringe who believe in privatised healthcare in this country (solid grounds for committal if you ask me), however the horrors of the U.S. Health system and its debt culture should be enough to put anyone off. I think that both Moore and Benn hit the nail on the head, that most of what happens in the U.S is about controlling an impoverished population.

I feel deeply sorry for the millions of people in the U.S who must be suffering in debt, poverty, depression and desperation. The more I read, the more I discover of a country where so few can have so much, and so many can have so little. I don't understand how they put up with it.

Is this really the American dream?

The French have free education, free health, easier working conditions, better living conditions, more productive people and "a government that is afraid of the people". In the U.S. it seems there is the opposite.

I hope we keep the government cowering in the UK, I'd like to see more more clear blue water between us and this American nightmare.

Dumping patients on street curbs, not treating their own heroes of ground zero, deciding which finger to lose on the grounds of which is cheaper - it's sickening indeed.

I can only assume that this film is exaggerated, because, I can't believe a nation of people who pride themselves on freedom would accept such slavery to poverty. If it is true, it is shameful.

Amazon vs Google

Today, there has been a lot of talk about Amazon vs Google in the cloud computing world. In my opinion Amazon could be facing a losing battle.

Why? To explain this, I first need to cover a bit of recent history. A couple of years back, the company I ran, built a utility computing service with an application development framework known as Zimki. 

You can think of it as an earlier GoogleAppEngine but with JavaScript not Python. Now, a key part of the product strategy was to open source the technology in order to encourage a competitive ecosystem of providers. This was designed to overcome the lock-in issues associated with Software as a Service. 

By aiming to create a competitive ecosystem, we were adopting an approach of a small piece of a big pie rather than a big piece of a small one.

The key to building an ecosystem is to have a standard language, a standard set of primitives and complete portability of code and data. For a developer, if your code uses these standards then you know it will run in any standard environment, no matter who the provider is. You also can switch to any other provider. To make this work required two things:-

1. A fast way for vendors to operationally implement the standard without losing strategic control of their business. This requires open source.
2. Some form of assurance and monitoring service, to ensure primitives were complied with and portability maintained.

Now, whilst Google hasn't provided their environment as open sourced, it has provided an open sourced SDK that "emulates all of the App Engine services on your local computer". This appears, though I'm not a python expert, to contain all the primitives and information needed to build a compatible environment to GoogleAppEngine. This allows for companies, vendors and ISPs to create competing but compatible systems. It's almost as if Google has offered a blueprint for a web operating environment and asked the rest of the community to come compete with them. It appears that way, because that's exactly what they have done.

I suspect that challenge will be taken up rather rapidly. It may not be the fastest way of creating a standard but it potentially will achieve the same result, especially if a credible competing open sourced version is created.

As soon as that ecosystem starts, I personally think it's curtains for any alternative environment that doesn't adopt such an approach. A market of competing providers is always more attractive to a consumer than a monopoly. There is one exception to this, which I've highlighted before.

Interesting times ahead. I'm becoming more impressed with Google's move and the SDK. Of course, I'm looking forward to the JavaScript version.

One last thing, if there are group of people out there suddenly thinking why don't we build a highly scaleable open sourced service which matches the standards outlined in the GoogleAppEngine Open SDK, please let me know!

--- 16th April 2014

In the end, there was little adoption and provision of an alternative GAE based upon the SDK other than the work of AppScale. Hence no market of alternative providers was built. However, the above game has been once again played (but much later) with Cloud Foundry which is providing a fully open sourced platform. This looks more promising.

Say it with pictures .... part II.

Following on from essential IT knowledge, I thought I'd bring you essential management knowledge in Venn diagram form:

Figure 1 - Understanding magic bullet solutions.
(click on image for larger size)

Figure 2 - The always right project methodology argument
(click on image for larger size)

Figure 3 - Understanding organisation, people and management.
(click on image for larger size)

Every journey begins .....

So Google has fully entered the web as an operating system space, with the release of GoogleAppEngine. They now have a powerful stable of Software as a Service (SaaS) offerings from specific applications to web services to a framework for software development. On top of this, they have Gears and Android which provide offline access to services and also a software platform for mobiles.

I do hope that at some point in the future, they will connect it all with a JavaScript everywhere type environment. There's an enormous amount of untapped potential here.

I see that Google have also open sourced the SDK of GoogleAppEngine. Whilst I applaud the move, I'm hoping that either Google or IBM release an entire open sourced stack and make a move to create an ecosystem of providers in a competitive utility computing market.

Why? Well, there exists a potential threat for a proprietary based technology to create an "open" utility market place which is instead an interoperability disaster. I'll be talking about this later in the year.

Companies should always be wary of any SaaS offerings which are not based upon an open sourced technology. I say wary because not only do you lock yourself into a platform but also you are unlikely to have multiple providers and hence a marketplace. Unless you have a plan which can cope with a major loss of functionality or data and has a valid second sourcing option, then you need to be aware of the risk you're taking. James has written a fair but robust piece on this.

That said, Google's announcement is a step .... I obviously want to hear more about the direction they are going in though.

Meeting new people ...

I gave a talk last week on innovation and commoditisation to a small group at dunnhumby. It was a great opportunity to discuss a number of concepts with experts from another industry. They certainly raised some interesting and very valid points.

Anyway, I've made a video of the presentation (it's a slightly modified version from the original and as usual I've had to record some audio with just me speaking at home. No atmosphere and without a proper microphone - the audio is a bit pants. I was also a bit tired, when I get a moment I'll probably redo this.).

It covers many of the usual subjects but with a few new additions. The themes are:-

• commoditisation
• commoditisation of IT (for example SOA, SaaS etc)
• innovation
• how stuff happens
• why nothing is simple in management
• the different life-cycles of an activity
• the need for different project methodologies
• the problem with outsourcing
• organisation and enterprise 2.0
• my research into business process modelling
• predictions for the future

Managing a complex world (45 mins)

20th October 2015

The original video was on blip which stopped their consumer service. So, I've reloaded the original to youtube.

The People Vs Company innovation ...

Activity (a process, sub-process or the result thereof) is an important concept, because the firms that we work for are organisations formed from such activities and the people, or actors, who perform them. Whilst it is easy to be seduced by the idea that an organisation is something concrete or real, like a building, in reality an organisation is defined by and only exists within the relationships between these actors and their activities. Take the actors and activities away and then, unlike a building, an organisation vanishes.

People come and go, relationships are forged and broken and activities change. Within any industrial ecosystem, new activities or innovations are a constantly emerging consequence of competitive life. Any innovation which is successful will be copied and the innovation will spread. Over time, given the right conditions, a once novel innovation, like the telephone, will become common and commodity-like. Of course, it doesn't have to be a product, it could be a process such as computerised payroll. Activities are on a constant conveyor belt from innovation to commodity.

The constant flux of activity and actors means that organisations are in a permanent state of change. Within this maelstrom, a company needs to constantly adapt as activities will continue to evolve in the outside world. This is the business equivalent of the Red Queen Effect from evolutionary theory, in which organisms must continuously evolve just to remain stationary relative to their competitors. However, standing still is not enough as innovation is a necessity for long term survival, as shown by Schumpeter's work on "creative destruction". To make matters worse, at the heart of innovation is a paradox between the order we need to exploit our current competencies and the disorder we need to create new ones.

Could it be any more challenging for a company? Why, yes;

"Innovation is a complex process [not static] produced by the interaction between structural influences and the actions of individuals."

Perspectives on Innovation in Organisations, Slappendel, 1996.

Hence organisational innovation is an interaction between the activities undertaken and the skill, knowledge and interactions of the actors who are part of that structure. In other words, organisations don't innovate, people do.

So in summary:

• change is a constant necessity in order for a organisation to remain competitive today.
• innovation is a constant necessity in order for a organisation to remain competitive tomorrow.
• an organisation only exists in the interaction between people and activities.
• organisations don't innovate, people do.

I mention this as someone said to me today, "our people are an important asset" and we're an "innovative company". I'd say that the former statement is a short-term approach to company survival and the latter statement is just plain wrong.