Agility Drivers

When the facts change, I change my mind. What do you do, sir?

— John Maynard Keynes

A management practice is mature when even government bureaucracies decide to adopt it. The March 2011 publication of UK’s ICT strategy marks this moment by advocating that “the application of agile ICT delivery methods [...] will improve government’s capability to deliver projects successfully and realise benefits faster.”. This begs the question: were we misguided during the decades we were advocating stringent control of requirements and a tightly milestone-driven development process? Interestingly, this was not the case. We were right then, and we’re right now. Things have changed, and this is why we can nowadays smugly apply agile practices reaping impressive dividends. Numerous new factors are driving agility by increasing our productivity. Our growing ability to swiftly put together sophisticated software affords us the luxury to listen to our customers, to try out new things, to collaborate across formal boundaries, to make mistakes, to redesign as we move along—in short to be agile. Knowing these factors helps us realize when we can afford to be agile and when not. (Hint: agile development of a plane’s flight control software from the ground up is still not a good idea.)

Technology ...

On the technological front two key drivers are the system software we can use: the operating system and the database management system. While I admire the conceptual clarity and simplicity of the 1970’s 7th Edition Unix, the truth is that a modern operating system distribution offers many facilities that transform projects that used to be a major undertaking into a weekend job. These facilities include standardized networking, inter-process communication, graphical user interface support, sophisticated typeface and 3D rendering, and load balancing across many CPUs. Database management systems (perhaps thankfully) have not advanced as much on the functionality they offer, but have become ubiquitous. If you’re carrying a smartphone, you probably have one or more RDBMS’s running inside your pocket. Therefore, if a program requires structured access to persistent data, it can readily call a database; developers no longer need to waste time laboriously crafting bespoke data formats and access methods.

One level up come libraries. Gone are the days when each self-respecting program contained one or more sorting subroutines and other “utility” functions. Now the 6th Standard Edition of the Java platform comes with no less than 3776 classes and interfaces. From an AbstractAnnotationValueVisitor to a ZipOutputStream, it’s all there. Other platforms, from Perl to .NET are similarly feature-rich. This makes many software development tasks a simple matter of gluing together existing classes. And if a class isn’t part of the platform, it is likely to be available as an add-on component. And if the task at hand requires a large data set (say the Earth’s map) or relies on constantly changing facts (think of flight information), probably someone will have developed a corresponding web service we can use.

This brings me to application interoperability. After a few false starts (remember DCOM and CORBA?) we seem to have sorted out how one application can readily communicate with another lying a continent apart. Without laborious prearrangements the Internet and RESTfull interfaces provide us hassle-free access to data ranging from the movement of stocks to that of tectonic plates. This paradigm is so prevalent that public-facing applications failing to provide a web service interface are perceived as crippled. Libraries (again) make using these services a joy. With less than a screenfull of code I can obtain a book’s cover image from amazon.com, send out a tweet to thousands of readers, and even use a SOAP interface.

A standardized presentation layer for our applications is also slowly evolving. While AJAX is by no means perfect as an application development method, it offers a neat way to run applications from any desktop, and the establishment of HTML5 is likely to improve things even further.

Programming languages have followed pace with our requirements providing impressive features that used to be the subject of academic research: threads, nimble type systems, duck typing, modules, metaprogramming, support for object-oriented and functional programming are only some of the goodies that allow us to churn out each day more sophisticated and powerful code. Further pushing the productivity envelope are scripting languages, which allow us to express in a couple of lines what used to be a student’s final-year project.

All these impressive technologies I’ve outlined come at a cost. They require resources to develop, effort to master, muscle to run, and patience to endure their (inevitable) shortfalls. Yet even in this regard we’re blessed with many countervailing forces.

CPU power clock rates and memory capacities have increased by many orders of magnitude. The first IBM PC came with a 4.77MHz CPU, a 360kB floppy disk, and 64kB of RAM. During the past decade, having satisfied the needs of the most demanding office worker, the resources available on a modern desktop now provide us the power to run all the wonderful facilities I’ve outlined and also the tools to conquer their complexity. These include IDEs that provide us refactoring support and online help for all methods of Java’s 3776 classes, optimizing compilers and runtime systems that make our readable code run efficiently, unit testing frameworks that take the fear out of refactoring, light markup methods like Javadoc that finally materialize literate programming, and languages with an interactive top level that promote experimentation and bottom up development. Tools, such as wikis, instant messaging applications, version control systems, and bug trackers smooth the collaboration with colleagues across the hall and around the world. In addition, various cloud platforms allow the rapid and scalable deployment of end-user applications and developer facilities at a reasonable cost.

... and the environment

Technology is not the only force driving agile development practices. Social factors are equally important. Software developers are no longer retrained mathematicians or physicists, but computer science graduates who are also increasingly participating in cross-disciplinary programs. It’s only natural to expect more from such people, including the ability to closely collaborate with clients on solutions that address their real needs. Changes in the management culture, where hierarchies have become flatter, reporting less formal, and teamwork de rigueur, also help shift the software project emphasis from the milestone-based hand-over of responsibility into collaboration with colleagues, in-house business and marketing departments, and customers.

Moreover developers can tap into a variety of resources for help. Programming-by-googling is becoming the standard way of writing code as developers use the web to locate code examples, suggestions for handling cryptic error messages, peer advice from professional forums, and, most importantly, open source code. This last resource is an agility enabler through the vast number of readily-accessible useful components coupled with their (sometimes deceptively) hassle-free procurement and licensing. Do you want to add PDF generation or speech output to your application? You can download and link such components to your code in a matter of minutes.

Then come two opposing forces. First the ubiquitous availability of user-friendly, polished and versatile consumer-oriented IT applications has increased the expectations of all end-users. They want their ERP to be as fast as Google’s search and its interface as intuitive as their iPhone’s. That’s a tall order for a traditional development method, and a call for agility. At the same time, these and other consumer applications are lowering users’ expectations regarding features and reliability. Millions are discovering that sending 140 character messages over a platform that occasionally overloads displaying a whale image is a perfectly acceptable, even addictive, way to communicate. Agile practices are ideal for swiftly delivering such good-enough solutions in an environment that can tolerate the occasional failure.

Finally, the ubiquitous availability of IT infrastructures makes the scale of today’s deployments considerably larger. IT support that used to target accounting and management is nowadays covering all the organization’s employees and customers. This increases the risk of centrally-planned, rigidly-executed projects, further driving the need for agility.

I’m sure that by now you’re convinced that the landscape of modern software development has changed vastly over the years in ways we haven’t yet fully appreciated. It’s therefore natural to adjust the way we develop software. Where agility drivers are present it’s a shame to preserve the status quo. Realizing that better outcomes are possible we must adjust our development processes, demand more from our software suppliers, and develop in-house capacity to organically grow applications and services that will delight and even captivate our users and customers.