When major government departments make decisions to spend billions of pounds they are more likely to bring in business psychologists after things have gone wrong than to ensure it goes right in the first place. Decisions are human processes and people have limited capacities however brilliant they may be. Psychologists understand this and can often identify the blind spots that are responsible for projects being set up to fail.

Yet the higher up the organisational ladder the greater the fear of falling, and top executives generally apply one of a few limited strategies. The most typical is to move on to another role, or retire before the consequences of poor decisions become too obvious. Another is to declare things a success, use draconian power to prevent the truth from emerging and to tough it out whilst all around are suffering. A third strategy is to take stock and try to learn lessons so that things go better next time

Project failure is so common in government that a poster on the wall of one office seems to capture the essence:

The six stages of a project
1. Enthusiasm
2. Disillusionment
3. Panic
4. Search for the guilty
5. Punishment of the innocent
6. Praise and honour for the non participants

It is good practice to review lessons learned, but the process can not be carried out properly by the people who were involved alone as they have formed their views and are more likely to justify the problems than dangerously reflect on their own limitations. Sometimes the lessons learned exercise is directed to safer areas such as what went right. Very few reviews engage with the real source of the problems - the minds and inter personal interactions of the people who carried out the project.

TSR International, a firm of business psychologists and knowledge engineers, was called into examine a massive IT based project and to propose improvements on the process used, make suggestions about managing complexity and report on lessons learned and how they might be applied in future.

The initiative was led by Nicholas Moore who has an accomplished career in large systems projects and Kevin Kingsland who brought special knowledge of individual and corporate learning. Both have extensive backgrounds in complexity science and draw on a related theory of psychology called Spectrum Theory.

The review was conducted by engaging with departmental Director Generals up to the Permanent Secretary. Rich qualitative evidence was gathered from this top group and from access to all relevant documentation. The most common testimony of contributors was a resigned and slightly depressed view that their processes were disconnected, there was a lack of common sense and that communication failed. Yet many of the reporters were dedicated and highly talented people.

How could such a group of highly motivated top talent produce such a mess? Surface reports provide the usual generic criticisms of failed projects. They are much like the practice of blaming things on a lack of time when really the word 'time' hides the underlying problems of lack of organisation, resources, critical path planning or prioritisation. Underlying large project failure the major problem is that people educated in an era of predominantly linear processes do not have the personal mental models that should equip them for non-linear behaviour of complex systems. Indeed, the very word 'complex' is lost on people as it is mistakenly taken to mean very complicated. It is nothing of the sort but refers to the technical behaviour of dynamical processes that behave in unpredictable ways.

The second particularly difficult challenge is that of interpersonal communication. Eloquent speakers and clear thinkers tend to think that they are good at communication. But people live in sometimes very different psychological worlds and very little real communication actually takes place. New kinds of psychological insights and skills are required to make diverse teams work. People who have different mental models of the world do not share common sense.

Technological, organisational and social change have moved so fast that few people have kept up. Top executives have got to refresh and upgrade their personal psychology if they are to perform and understand how things go wrong, and more importantly how things can be optimised to go right. Calling in business psychologists, who understand complexity and communication, at an early point can save vast amounts of money and heartache.

So what did the department gain from this late intervention? Many of the recommendations made have been implemented and there are claims that the department has learned its lessons. They feel they got their monies worth. We, on the other hand, are not quite as convinced. It is important that tax payers get their monies worth, not just in an initiative like this but for the future. Until top executives can raise themselves above their own careers and accept that we are all in a life-long learning situation, and commit themselves to "hurting their own heads" which is the necessary hallmark of true new learning, we are going to have many more failed projects and exhausted, disillusioned people.