Thursday, February 13, 2014

The slow but overwhelming flood of progress

Back between 2004 - 2005, development in Fotango (a company I used to run) had improved significantly through the efforts of Artur Bergman (now CEO of Fastly), James Duncan (now with UK Gov) and a host of others.

1) our infrastructure was provided by a virtualised environment known as the Borg. A mass of racks providing standardised virtual machines (i.e. 'commodity' infrastructure provided with Xen) all controlled by the Borg Queen who's job it was to create virtual machines on demand, configure them (we happened to use CFEngine), install them and monitor the applications and estate health.

2) we'd been using test driven development for a considerable amount of time and had built extensive test scripts for our applications. 

3) the process of delivering applications to live was relatively simple. A developer pushed an application from the code repository (we used subversion, Chia-lang Kao used to work for us) to production, the test scripts confirmed the system was in an acceptable state and the Borg Queen took over. Part of the configuration files determined what other services were consumed, the destination of the system and it managed graceful replacement and rollback if necessary. We had nightly rebuilds and validation of the entire estate.

4) the organisation itself was broken into three core groups which had started with IT. We had development (pioneers) who created the novel. Frameworks (settlers) who identified common patterns for provision of new web services. Systems (town planners) who managed the estate in terms of core systems and services (i.e. Borg, testing agents, monitoring agents etc).

5) we recruited from around the world and mined open source communities (especially the Perl world) for talent. LPM used to even call us the Borg as we assimilated so many ... well, it's your own fault for being so good.

6) the company had extensively used web services for many years, most of the systems ran on web services and we'd even started experimenting with the idea of providing a more extensive list of public web services. Later on in 2005 this became our utility platform service known as Zimki, one of the first public PaaS environments which provided a server side Javascript development environment with common services (nosql like object store etc), migration between different Zimki installations, automagic translation of functions to web services and even billing information down to the function. The closest you would come to Zimki today would be a combination of CloudFoundry and the Node.js buildpack. The vision behind it was all 'Pre-shaved Yaks' and there are long list of people to thank for its creation, especially Tom Insam (who later built doplr and lanyrd) and Mark Fowler.

But even in 2005 we had our own legacy IT, a lot of horrors that remained from the past such as an ill fated SAN effort and a tortuously complex past platform.  We'd also gone through some painful lesson e.g. agile everywhere. However, we weren't alone with this and we continued to improve, implementing by end 2006 a more open working environment, hackdays (every other Friday), agile design (paper prototyping), BYOD (in fact we had a store cupboard of help yourself macs in case you needed it), events (opening up our Old Street offices to the local tech community ... this was back in the days before Old Street became the technical powerhouse it has) and a host of other techniques (including mapping).

For us, all the above was just normal, it was how stuff was done. So why do I mention this? 

Well, today all the rage is about continuous deployment, design for failure, cloud, server side javascript, PaaS, nosql, open source, BYOD and agile techniques. Whilst these fields have progressed extensively in the last decade, it always surprises me to hear how many companies are so far behind the game. It does appear that you can measure the delta between the leading edge and the laggards in terms of decades of technology. It's like Enterprise 2.0, a coined term in 2006 by Andrew Mcafee to describe a set of technology changes that Euan Semple was busily implementing in the BBC before 2001 and for which some companies are only just getting started today.

However, the really fascinating part of this is how the changes tend to be exponential and create a punctuated equilibrium with the past. So what starts as a trickle becomes a flood. That's the thing about cloud, devops, big data, enterprise 2.0 and all these related topics. Today is not about the beginning of a subject but the overwhelming flood that you have to adapt to. Alas for some companies it's already too late.

Out of interest, it was the work we did back then that enabled me to develop the models of how organisations evolve and later test this was happening.  I mention that because Matt Asay's article on the Future of DIY IT is spot on. That flood is upon us and 'traditional IT is dead. Not just a little bit dead. Dead dead.' 

Ok, technically it won't completely disappear (things rarely do) as it'll whimper on in some niches for quite a time. There's still niches for modern day swordsmiths but they're niches. Have no illusions though, the flood is here and like it or not, you're going to have to change.