So What is the Actual Problem?
In a word, management. This and the fact that these kind of institutions, private and public, are political in nature. As such, management like to divorce themselves entirely from the promises they make, their mistakes and their responsibilities. There is no accountability for failure, and failure is almost always repackaged as success.
HyperNormalisation and "Management by Exception"
The word HyperNormalisation is taken from the book "Everything was Forever, Until it was No More: The Last Soviet Generation", Alexie Yurchak 2006.
Put simply it describes the state of the Soviet Union during the period prior to its collapse. Despite the fact that everyone knew the system didn't work, they maintained the pretence that it did.
Ultimately even intelligent, well educated citizens came to believe that there was no alternative, that the political packaging was in fact reality.
HyperNormalisation is a disease that has spread to many large organisations, an inevitable result of bureaucracy before product and management by committee.
Designed for Failure
Prince II for example, a well known approach to project management, a standard in the UK public sector in-fact, practices "Management by Exception". Intended to free up managers from, you guessed it... managing.
This approach is a disaster, regardless of its intent. A self-cancelling mechanism with no impact other than to separate management from responsibility. Simply kicking off a project, burying your head in the sand and assuming that all's well unless told otherwise doesn't actually work.
So why is this even necessary? Because managers are too busy to manage. Why are they too busy to manage? Because the way they manage in unmanageable. Perhaps a bit harsh, but the fact remains that Management by Exception is entirely incompatible with the British culture.
Subordinates will do whatever it takes to hide failure from their bosses, and in turn the bosses don't want to admit that a project is out of control.
In such environments, where networking has become the primary survival mechanism, it's become politically incorrect to raise concerns, highlight failure or state the obvious. Group-think has degenerated into blind obedience and those that do point out that something is wrong are scorned and pushed to the outer edges of the flock.
So how can Management by Exception work if raising an exception is so frowned upon?
Obviously it can't, the entire concept is worthless. If so much effort and resources are being dedicated to techniques, such as Prince, Lean, Service Excellence and even Agile, then why are these institutions so incapable of delivering?
Because they are not addressing the underlying issues:
- They are not using the right people to do the right things.
- They don't know enough about the size or specifics of the job.
- They don't understand how the job can be turned into a solution.
Love Me, Love My CV...
This bit explains why a lot of large projects are either completely unnecessary or should actually be small ones. I'm going to go on about it a bit as this is a cultural issue that has a far reaching and very negative impact in almost all organisational areas, not just software delivery.
Warning: The following may be considered controversial. By clicking "I Don't Care, Show Me More" you are agreeing that a) this is between us, and b) that I won't be held accountable if you get unexpectedly disconnected from the Matrix.
A Scary Story...
Many years ago I may, or may not, have been working for a very large client. They were, at the time, undergoing a group wide initiative to rationalise and consolidate their systems and processes. Sounded exciting at first and it was huge, the budget was staggering.
One of the key areas the project was attempting to address, was the organisations heavy reliance on large numbers of EUCs, or End User Computing solutions. These being mostly the spreadsheets and databases developed, outside of IT, to address the limitations of the core systems. The culture of the organisation was such that EUCs were considered bad.
Given the complexity of the organisation, it seemed obvious that an off-the-shelf solution simply didn't exist. Even if it did, it wouldn't be long before, somewhere in the world, some sort of market or regulatory change would require a work-around.
Paradoxically, despite their promise to do away with the need for EUCs, during delivery of pretty much every big solution, gaps were being identified and EUCs were being developed to plug them! The "no need for EUCs" was a simple sales pitch.
Interesting, very interesting...
Clearly, given their history and the experience of their head-office management teams, the best they could hope for was a solution that was relatively flexible and open. And, being realistic, that it would be supported by as many EUCs as necessary. Over time these would either bed in and be formalised, or dropped if the gap was plugged or the need removed.
It seemed odd therefore that they would be spending hundreds of millions every few years attempting to achieve the impossible.
I asked one of the bosses what was going on. Wouldn't they be better off simply formalising the EUCs every couple of years, ensuring they were still necessary, adequate, secure and documented? Feeding back to the software provider the solutions gaps and limitations that they might be addressed?
Perhaps, coming to terms with the fact that nothing does everything might be a good place to start?
The response I got was a little bit disturbing. She told me that it wasn't about what was best for the organisation, it was about what looks good on peoples CVs.
To have a CV that listed instances of implementing high cost change or solution delivery, managing budgets of millions and even hundreds of millions, was far more important than being able to say something like "HR Manager" or "Customer Services Manager".
Essentially, most of the change and expense was incurred simply as a way to make people look good on paper. It wasn't important how well they delivered, or even how good or cost effective the deliverable was. What mattered were a) budget size, b) head-count and c) buzz words.
The impact of this on the organisation was, and still is, disastrous. They still go through this cycle every few years, spending vast sums, and have never delivered a solution that is complete, or hasn't been superseded by another.
This doesn't just apply to the systems themselves, but also to the management methodologies intended to deliver. They have moved from one to another without every truly implementing them or realising any benefits. All this for no reason other than someone high up wanting to be able to say "I delivered Lean", or "I delivered Agile" or most ironically "I delivered Service Excellence".
But Hasn't Anybody Noticed?
Theoretically, no... The very same organisation displays three traits that can only be described as bewildering. Each exists solely to fool the people so far up the tree they are rarely seen on the shop-floor (so to speak). To create the illusion that all this waste is both necessary and a great investment in the organisations future:
- They reward incompetence: For example, during a Lean initiative incentives were offered to the teams that identified the greatest time savings. I.e. the ones that were so bad they weren't already operating efficiently. It was like a form of amnesty, only necessary because the people in charge had no idea what was going on in their own areas. None of the Lean fundamentals were delivered. Bad practices continued, nothing was learned and continuous improvement was unheard of. Despite this, most of the people involved claim to have successfully delivered Lean.
- They offer incentives to feign approval and success: For example, following delivery of a large, and completely unnecessary, internal (social) networking application, people were offered incentives to use it. In other words to waste time playing with an application that didn't contribute anything to the bottom line. And... wasn't nearly as funky or entertaining as other on-line time-killers such as FaceBook and SnapChat.
- They lie about benefits delivered: As a developer this is extremely annoying. In cases where a new initiative is not quite delivering the benefits promised, they credit time savings from something unrelated. For example, a large complex solution that delivers savings of a hundred hours a month, but uses Robotics to simply download a file from a Unix server and copy it onto a Windows one, saving just five minutes, has all the credit assigned to the Robotics team.
The Emperor is Naked!
Over the years the same people, and consultancies, that promoted and "delivered" the old methodologies are the same ones seen peddling the new ones. Often making as much, if not more, from delivering changes to the way we manage as they do from delivering actual "solutions".
Nobody is asking what was wrong with the old way. The really scary thing is, it's not important. What's important is that nobody is acknowledging the same people, or types of people, are failing, over and over again, to deliver solutions and methodologies that even make it out of adolescence.
Instead of addressing the problem, namely the people and the culture, they repeatedly change the approach in the misguided belief that it will compensate for their inability to deliver.
A correctly bedded in methodology, or a long-term flexible solution would be a disaster both for CV and business.
Process before Deliverable
Primarily, it's the delivery of the technique that has become the measurable deliverable, not its effectiveness or even the degree to which it's followed. Delivery is measured in smiley faces, networking events and pretence.
Management fads come and go, all delivered as roaring successes and then become obsolete within just a few years. So if they're so good, why are they superseded so quickly? Because, either they are not appropriate to the context or not actually implemented effectively.
Failing to Measure by Results
All this might sound harsh but the fact remains, the cost of delivery is wildly out of control, are time-scales and the quality of what's being delivered. Worse still, the people responsible are getting away with it and going on to do even more damage! The problem is that in many of these large organisations it has become politically incorrect, as it is in government, to hold senior people responsible for their actions.
Look at the state of the deliverable during, and at the end of, a project. Is it what you expected? Are there so many limitations that you have dozens, even hundreds, of off-shore data processors massaging the data every month? Has this project triggered a raft of smaller projects, manual processes or risks? These are all symptoms of a bad delivery.
Management fads, for example Agile, DevOps and Service Excellence, even Lean, are not the problem. They all have their place and if worked correctly can deliver enormous benefits. So why are they so frequently superseded? The fact is they are rarely implemented properly. And even if they are, they don't necessarily guarantee quality. Rather they provide a delivery approach that, if followed correctly, removes accountability.
We could blame process and committee, but at the end of the day it's far simpler than that. The primary reason things are allowed to get so out of control is that they are allowed to get so out of control. Nobody is jumping up and down shouting "This is wrong, it's just not good enough!" or, "Who put this Muppet in charge".
An Absolute Must!
I mention this separately, and will repeat myself, because this one thing solves MOST of the problems associated with project delivery... Completely separate administration, development/delivery and compliance (see Image 2 below). This way the right people will be doing the right jobs, they won't muddy each others paddling pools and the overall quality of the deliverable will increase exponentially.