Inevitably, everyone considering process improvements of some sort wants "metrics", in other words, some statistics that will "prove" that the path down which they are about to embark will result in improvements. Of course the phrase "your mileage may vary" comes to mind - no matter how successful someone else has been with a particular approach is no guarantee that others will achieve the same results. In the end, all they really tell you is that someone else has done it, so there is a possibility that you can do it, too.

Why are metrics so hard to come by, especially metrics that plot improvement? The biggest reason is that most organizations that embark on improvement efforts don't measure their initial performance. While they know that things are bad, they often can't quantify them beyond a vague sense that "things take too long" or "we have too many defects". While these hunches may be correct, the lack of data makes it difficult later to quantify the benefits achieved by an improvement effort. Often the improvement effort itself will introduce some measurements, so benefits achieved in later improvement cycles can be quantified, but the benefits from the early improvements (where the improvements are often the most dramatic) often remain largely undocumented.

This lack of metrics creates a couple of problems. First, it makes it all but impossible to compute the ROI of the improvement effort. Without before and after measurements, improvement results are largely subjective and anecdotal. The lack of demonstrable data makes it hard to justify further improvements, with the result that improvement efforts can stall without hard data to support the effort.

Second, it makes it hard to understand where to focus improvement efforts. There is a saying that you can't improve what you can't measure, and so without data it is very likely that incorrect diagnoses may be made. Tying back to an earlier topic about desired outcomes, we need to be specific about the desired outcomes. Saying "we want to reduce the cost of carrying inventory" is vague, since we could solve the problem in a number of ways: we could contract for cheaper warehouse facilities, we could reduce the level of inventory, or we we could increase inventory turnover (the rate at which inventory is depleted and replaced), which reduces the cost of financing the inventory. Or we could do all three. If we don't know how many times the inventory is turning over, we will not be able to set appropriate goals for improvements, causing us to waste time developing the wrong solution.

There is a great book by Henry Petroski that everyone developing software should read: To Engineer is Human. Petroski analyzes some famous engineering failures and talks about why engineering is so difficult. While not directly about software, the intellectual process of engineering structures is quite similar to what happens in software; if anything, it is more mature and well-founded on scientific principles.

There's an interesting review of Petroski's book at http://www.csd.uwo.ca/staff/magi/personal/humour/Computer_Audience/To%20Engineer%20is%20Human.html that relates the book to software development.

The key take-away for me is that there is a lot to learn in analyzing failures. Many years ago I did a lot of performance benchmarking and performance tuning. Analyzing the things that caused performance to become bad taught me a lot about how to develop systems that were more resilient to increases in load and how to recognize performance bottlenecks in the design.

It's also useful to dispassionately analyze the reasons for project failure. When projects go bad, there is a strong tendency to either engage in the blame game, or to quietly slink away from the project, hoping no one notices the bad smell emanating from it. Projects rarely fail due to "bad people" working on it; many failed projects are doomed from the start by bad management, unrealistic goals, poor team interactions, etc. Understanding these failure modes help the organization to learn and improve. Sometimes the lessons are hard to accept, but you have to face your shortcomings before things will start to get better.

There's a great quote in The Mythical Man Month that goes something like: good judgment comes from experience, most of which comes from bad judgment.

There is a great book by Henry Petroski that everyone developing software should read: To Engineer is Human. Petroski analyzes some famous engineering failures and talks about why engineering is so difficult. While not directly about software, the intellectual process of engineering structures is quite similar to what happens in software; if anything, it is more mature and well-founded on scientific principles.

There's an interesting review of Petroski's book at http://www.csd.uwo.ca/staff/magi/personal/humour/Computer_Audience/To%20Engineer%20is%20Human.html that relates the book to software development.

The key take-away for me is that there is a lot to learn in analyzing failures. Many years ago I did a lot of performance benchmarking and performance tuning. Analyzing the things that caused performance to become bad taught me a lot about how to develop systems that were more resilient to increases in load and how to recognize performance bottlenecks in the design.

It's also useful to dispassionately analyze the reasons for project failure. When projects go bad, there is a strong tendency to either engage in the blame game, or to quietly slink away from the project, hoping no one notices the bad smell emanating from it. Projects rarely fail due to "bad people" working on it; many failed projects are doomed from the start by bad management, unrealistic goals, poor team interactions, etc. Understanding these failure modes help the organization to learn and improve. Sometimes the lessons are hard to accept, but you have to face your shortcomings before things will start to get better.

There's a great quote in The Mythical Man Month that goes something like: good judgment comes from experience, most of which comes from bad judgment.

Fred Brooks’ book The Mythical Man Month is still a great read after all these years, and still contains lots of real gems. One that I was reminded of recently is that there still is no “Silver Bullet” technology or technique that will provide guaranteed, foolproof results. Making improvements is still hard work, and the secret to success lies mostly in execution. A mediocre plan well executed will beat a brilliant plan poorly executed every time.

In the course of my work I often consult with companies considering using iterative development approaches, and they often want to see statistics about improvements that other companies have achieved. At one level there is nothing wrong with this – it is natural to want to know that a particular approach has been proven in practice. At the same time, it’s important to understand that iterative development, like a host of other “best practices” embodied in the Rational Unified Process (RUP), is not a “silver bullet”. Success or failure requires lots of hard work and the techniques alone are not a guarantee of success. Good execution is essential.

In the best case, what best practices provide is a foundation on which to base the execution. They can reduce the amount of time spent figuring out what to do by providing a source of ideas for solving various problems. They are not, however, a prescription for success – good implementation is still the essential ingredient. Often, we find that in addition to helping organizations adopt best practices, our largest contribution is in helping them to execute better. The best practices serve as a vehicle for facilitating this, but alone they cannot achieve the results. After more than 30 years, there is still no “Silver Bullet”.

Delivering successful solutions requires giving people what they need, not what they want. The conventional approach to problem solving is to ask people what they want and then give it to them. The problem with this is that people often fon’t really know what they want. In addition, asking someone to tell you what they want is basically the same as asking them to solve the problem. In the case of technology solutions, this often results in less than desirable results.

What we need is a way to get at what the stakeholders of the solution really need. The problem is that when you ask them what they need, they still just tell you what they want. There is a technique that helps to cut through this. I first learned about this a couple of years ago, from an article in the January 2002 issue of the Harvard Business Review entitled “Turn Customer Input into Innovation”. The technique centers around getting people to focus on “desired outcomes”. In other words, get people to describe the end result they would like to achieve.

Focusing on desired outcomes helps move everyone beyond defining “requirements” to end results. The best way to do this is in one or more facilitated sessions in which the stakeholders are encouraged to describe what results they would like to achieve with the solution – desired outcomes. Armed with this information, development teams are free to be creative, exploring solutions that the stakeholders may have not even imagined. In some cases, these new solutions may be not only better solutions but also cheaper solutions.

Development teams are sometimes weighed-down by the sheer mass of requirements for a system, many of which could be simplified or replaced if everyone had a clearer idea of the desired outcomes that the solution needs to produce. Similarly, focusing on desired outcomes frees the stakeholders from micro-managing the system development through the requirements, but also gives them control by providing a way to establish accountability for results. In short, everyone wins.

Einstein is said to have observed: "Perfection of means and confusion of ends seem to characterize our age." This is a variation on one of my favorite quotes, attributed to Thoreau, "The elevation of ends and the simplification of means is the goal." And then of course there is Eric Sevareid, the late journalist, who observed that "the chief cause of problems is solutions". Clearly, we have a problem with understanding the problem.


These perspectives shed some light on what is a fairly large problem in development projects: they tend to be in such a hurry to develop a solution that few pause to ask "a solution to what?". When someone comes to us and asks us to build something, especially when they come with a handful of money, we are likely to take them at their word and build what they say they want. But sometimes (or often) this isn't quite right.


We have been conditioned by conventional wisdom to accept that people know what they want, even though our experience is more like "people don't know what they want, but they know what they don't want when they see it." If it was cheap and easy to develop solutions we might take a more Darwinian approach - develop a lot of different solutions and see which ones survive. On an economy-wide scale this is kind of what happened in the late 90's with the so-called "internet bubble" - lots of solutions to problems that were poorly understood or, in some cases, non-existent. But solutions are not cheap or easy to develop, so we need a better way.


Conventional wisdom suggests that we can find the right solution by documenting the requirements for that solution. But wait a minute: requirements are descriptions of a solution, not descriptions of the problem. If we have misunderstood the problem then even the best set of requirements will not help us. At best, good requirements reduce the cost of developing a solution by reducing ambiguities; they don't help if the people giving us the requirements are not describing the right solution.


While it's convenient to push off the problem onto subject matter experts and say that it's their problem to understand the problem, at the end of the project if the solution doesn't address the problem then the failure reflects on everyone. In addition, understanding the problem often triggers creative solutions. So understanding the problem is the real key to better solutions, not just better (unambiguous) requirements.


So how do we better understand the problem(s)? I've been doing some thinking and reading about this, but more on that a future posting...