Introduction, Context and... The Elephant in the Room
In order to really grasp what it takes to successfully develop and deliver large IT solutions, you have to first come to terms with the fact that it's simply a job of work. Just because the subject matter is complicated doesn't mean the delivery has to be complicated. In fact, most failures can be entirely attributed to the delivery, not the problem requiring a solution.
As well as describing how to avoid many of the pitfalls, this article focusses on why so many large IT projects cost such vast sums of money, and yet fail to deliver anything more than a token gesture. This is not a criticism of delivery methodologies as such, rather a discussion of the things that need to change in order for them to work effectively.
There are however various criticisms aimed at the people and cultures responsible, it is what it is. Failing to acknowledge the root cause of so many failures is the reason they keep happening. The solutions are simple and apply in many areas, not just software delivery.
The assumption that an organisation can change methodology and every thing will get better is entirely wrong.
What is it about IT Solutions?
We've all heard it a hundred times, but... For any large organisation, the ability to effectively deliver new solutions and process improvements is crucial. Costs can run into millions, even hundreds of millions, and the consequences of failure can be catastrophic.
Regardless of delivery model!
In many business sectors, the delivery of IT solutions has become a bandwagon. A bandwagon driven by witch doctors peddling snake-oil and fairy tales. With so much at stake, and such large budgets, it's easy to see why there are so many hangers on.
The fact is the largest costs, administrative and procedural, add no value.
Solution delivery has become a religion when it should be a science. Buzz words, empty promises and stories of pending doom and great rewards manipulate non-technical people into becoming followers.
The following (definitely no pun intended) will explain why this is happening, what's going wrong and how to fix it by spending less yet delivering more in less time.
Context, Scary Robots and Evolution...
Even today, most large organisations are propped up by fragmented systems. Systems that don't lend themselves well to integration or change. Systems supported by undocumented workarounds, waivers and manual processes.
Automation can sometimes offers a temporary fix by reducing manual effort. However, if a process can be automated, it should have either been part of the original system, or the system should be flexible enough to adapt.
Having an additional layer of technology is not necessarily the answer. It goes against "simplification" and has it's own overheads, weaknesses and skill requirements.
How much harder will it be in a few years to replace such a system? The automated jobs may well act as a type of documentation, but who will have a clear enough understanding to accurately describe what's being automated?
Ten years down the line, do you really want to be running an incomplete, inadequate and inefficient system held together by workarounds and robots?
Ultimately for most large organisations, in any environment, an adequate system is one that is flexible and can evolve. Constant change in markets, regulatory requirements and technologies make off-the-shelf solutions incomplete before they've even had a chance to bed in.
IT departments are so stretched and bureaucratic that in many cases solutions are delivered twice. First as a temporary "End User Computing" (EUC) solution, delivered outside the organisations IT department, and then by the IT department itself.
Sound a bit silly? Of course it does. Not just that but, many of these solutions could easily be delivered by a single developer in a fraction of the time.
Watch this short YouTube clip to see a light-hearted depiction of the type of people that shouldn't be allowed anywhere near management but invariably are. Particularly in financial services.
From the immortal genius and forever "out to lunch" Douglas Adams.
Part of the problem is cultural. In the UK for example, organisations are led by "managers". In the most part non-technical individuals who are essentially just administrators with authority.
Business games run in British universities measure results not by who made the most money, but by who did the best paperwork. Large organisations suffer from the same problem when it comes to systems delivery. Quality is measured not by the end result, but how it was delivered.
The evolution of bureaucracy lead to administration being confused with leadership. With that confusion came the pretence that if we administer and follow a process correctly, it should deliver. If it fails it's not our fault, the process is to blame or it's just one of those things. Either way we all did our best, so let's throw a jolly big party and celebrate the fact we made it to the end.
Another fundamental problem is politics and the way blatantly stupid decisions can be validated with the line "It's politics, best not to worry about it...". In this way the efforts of brilliant, dedicated and hard-working people are being quashed every day.
A conspiracy theorist might conclude it's actually corruption that's being packaged as politics, occasionally they might be right. But the truth is much scarier and far more controversial.
Largely office politics is just a wrapper for stupidity, bias, inability and inefficiency. Tragically, the people most capable of producing good results have been conditioned to accept this without question.
Management and Administration
Though it's easy to blame the "management", it wouldn't be entirely fair. Managing a project or, to be more precise, administering a project is a task that shouldn't be underestimated. However... allowing administrators to be responsible for delivery is both a mistake and dangerous (more on this below).
In the same light, expecting technical resources to be involved in administration, or answer to administrators is asking for trouble.
Imagine sitting an exam and every fifteen minutes being interrupted by someone asking "How do you find the questions, have we done a good job with the room?", "What's you approach going to be on the bit you haven't done yet?" or "How long do you think it will take to answer a question you haven't read?". Or more commonly, "What do you think this bit means?".
These are the kind of interruptions developers deal with every day. The more complex the project, the more time we spend explaining what it's all about to people who don't need to know and aren't contributing to the actual deliverable.
Writing and designing software is a highly sophisticated skill that requires a great deal of knowledge, experience and concentration. More so than most other professions, yet coders and analysts are often treated as if they had nothing much on.
Large projects focus most of their resources on things like managing expectations, morale and keeping up appearances. Requirements are often tardy at best, yet the assumption is they can be thrown at a developer and simply coded up, as if by magic!
Add to this the fact that almost all non-technical managers require constant reassurance, hand-holding and validation, and you get an environment that is entirely detrimental to concentration, creativity, logic and efficiency.
The fact is, the most likely cause of burnout in technical resources is the fact they have to deal with non-technical managers and administrators. It's exhausting.
Mixing Administration with Delivery (The Elephant in the Room)
This is without a doubt the most costly mistake in any delivery model. Despite the current trend to hunt down tedious and costly administrative tasks and either automate or do away with them, nobody seems to have learned that administrators, as managers (for example accountants), are the reason we got into this mess in the first place.
The majority of managers are still nothing more than non-technical administrators, and the vast majority of their efforts are directed at administration, packaging and office politics. None of which facilitate or add value to delivery.
Reasons not to Mix Administration with Delivery
- Managers rarely appreciate the complexities of software development or data structures, yet they insist on interfering in both.
- Developers have better things to do than helping keep Managers up to speed.
- People in general, not just Managers, get particularly stressed by things they don't understand.
- People in general, not just Developers, get particularly stressed by constant irrelevant interruptions and trivia (like administration).
- Administration, in most large organisations, is bloated and inefficient. Sound controversial? Ask yourself then, how is it that most large organisations are repeatedly spending the majority of their process improvement budgets on administrative processes?
- Accountants and "managers" are better suited to ensuring compliance and "doing" administration than they are delivering complex solutions.
- Hugely complex issues can be resolved in a single sitting when only a good Technical-Lead, Developers and real Subject Matter Experts and Analysts are involved. Bringing anyone other than customers into the mix invariably results in noise, confusion, pointless compromise and numerous counter productive follow-up meetings.
- Conversations like, Manager: "How long will it take?", Developer: "What do you mean by 'it'?" are reduced.
- Unnecessary milestones and objectives, like the ones that result from the above conversation, don't have to be managed, documented or maintained.
- Developers are blunt, if something isn't good enough they will likely say so. Administrators seek compromise and are frightened of approaching customers with problems.
- Managers recruit the wrong people, they surround themselves with people that make them feel safe.
- A single person who knows what they're doing, understands the scope of a problem and the context from which it originates, will always make a better decision than a committee of "interested" parties.
Administrators, given a bit of authority, frequently implement ridiculous policies and procedures.
For example, having to fill in three time-sheets. One for an agency, one for the organisation and another for the project. Each being authorised by a manager and forwarded on to another system for processing. I've seen this more than once!
To describe this as a monumental waste of time simply hides the fact it's actually just plain stupid.
Even if there are three requirements for that information, let the developer do one and have a cheaper resource, or automation script, do the others. Better still, let the administrators do it all. Even better again... acknowledge the person, or people responsible are actually stupid and do something about that.