It’s not rocket science

How is it that people work hard for long hours, and yet projects fail? Why do project debriefs become bitter tales of one team thwarting another? How can we be doing our best, and yet see it fall apart so spectacularly?

We now know how some of the mismeasure, frustration, and sometimes despair arises as we try to manage that relatively recent invention; the serial process – the manufacturing process. We recognize, often intuitively, that we are trying to optimize according to some reductionist/local optima approach but that we are actually bound by the process dependency and variation of our system. But what does it mean to optimize? Surely this means to do our very best. In fact, for everyone to do their very best. (From Dr K. J. Youngman’s website.)

In software development, we have a virtual shopfloor – a cyber pipeline – through which all work flows. If we all “do our best”, the pipe soon gets blocked with incoming work (if the account management/new business teams are any good) and the production teams don’t have the room they need to give each project the time it deserves; the creative team never fully hands off designs to the developers, as they always have one final tweak to make (after all, as they see their ideas realised, so they gain more ideas about them).

Organising the pipeline
Given that a chain is only as strong as its weakest link – that a group of boy scouts can only walk in single-file as fast as its slowest scout – any production process can only move as quickly as its slowest part.

If the new business team is fantastic, and concentrate on doing their job to the best of their ability, but define “best” purely in terms local to them (i.e. “I’m doing my best when I bring in most new business”), they will soon swamp the next team in the pipeline. If creatives never fully hand off their work, the next team can never really get started on it.

A brief detour to look at production problems
I’ve recently witnessed a massive project screw up, based on obstacles arising that surely were avoidable. A quick list of the major problems:

• Lack of clarity over what was to be delivered from the outset
• Changing requirements throughout the project
• Deadlines imposed from above
• Frequent personnel changes
• Deceit between different parts of the team over progress
• Command/control behaviour

Wow, where did all those problems come from?

Well – first among equals – the role of the new business and account management teams in bringing new projects into the company is where the rot starts (there are plenty of other places too, but this is the most common, possibly only because it’s first in the line). Their job was to bring this new client on board, and make them so happy with us that they would line up lots of projects as quickly as possible. And they did this job extremely well.

The team did their best, within the limited perspective of their role. But if we judge their work from the point of view of the production system as a whole, we see a markedly different story. One in which their contribution to the project’s failure is clear. Their priority was to ensure that the project delivered was exactly what the client wanted, so they changed requirements and delivery dates throughout the schedule. Nothing signed off was sacrosanct, and work had to be re-done on a regular basis, without changing delivery deadlines.

Although I know this sounds like finger-pointing, it’s not, I promise. I can do this same exercise with every team that played a part in the project, and each and every time doing “their best” will end up contributing to failure.

To prove that point: the development team also did their best, by demanding water-tight specifications. When they finally got them, the deadline was so tight that they worked extremely long hours (like 16 hour days for more than 7 weeks); they coded until they could no longer speak. They were demons!

But they failed to deliver. They blamed uncontrolled change and lack of specs, and late sign-off of designs. They tried to cover up basic mistakes in the bespoke framework (like writing a bespoke framework – anyone heard of PureMVC?), and didn’t test their code as they went.

And again: the project manager tried his best, within the definition of doing his job according to those who employed him. However, nothing was delivered on time; less than a quarter of the project was actually delivered; it was buggy and immensely expensive. It had failed on time, cost, and quality counts. And yet, I’m sure he would say that he did his best.

So, what’s going on?
Consider how we work: we are social animals that collaborate for a common good. Our social network, however, is fluid in terms of strong and weak links and thus we have many strategies for dealing with changes within the network – we can help each other, we can withhold support from each other, we can circumvent weaker members, we can break links and make new ones. Thus, although there is huge dependency in a social network, it is loosely-coupled and dynamic – everyone cooperates, with a few subordinating themselves to others at any one time*.

* by “subordinate”, I mean work to the betterment of others rather than oneself, put others’ needs and desires above one’s own.

Another form of experience also exists, though – our personal experience as individuals: here, we don’t need to subordinate or coordinate; we are free to improve and work to the best of our ability independently of those around us**.

** putting our own needs/desires first, I’ll be calling “maximising” later, and in regard to process, I mean promoting that particular piece of the process.

Now, the industrial mode: well, you might think that it’s a network, especially as workplaces have personal networks, but actually it’s far more linear: there are few if any alternative routes through the process (get new business -> specify the work to be done -> design -> architect -> implement -> test -> deploy -> review, for my industry). There is far greater dependency between the parts of the process – we can’t just ignore or cut out design because the designer’s on holiday. Finally, there is always one part of the process that is weaker or slower than the others. Improving the workflow of the other parts won’t speed up the total throughput because of that bottleneck.

In an industrialised line the only place that maximising adds value is in the bottleneck, and everything else has to subordinate to that bottleneck, which experience is almost the exact opposite of the social and individual experience, where the bottleneck can be avoided. Does that spell doom for our chances of success? Thankfully not, but we do need to change the way we work.

There is much within an organisation that contributes to a silo effect: there’s me, my team, my division, managers, department managers, division managers – all of which put an individual or group focus on working practices. But this can mask the strong serial dependency of the process pipeline. We need to view the system as a system, not as a set of silo’d groups. However, many of our measurements revolve around localised views of parts of the system, including our individual definitions of “doing our best” .

If each team involved in a project does its best, the project still runs the risk of failing (sometimes monumentally), so what should we do? Find out what is the best thing to do, and then do that to the best of our ability.

More anon.