The real reason so many software projects are late, over budget and cancelled.

•May 5, 2007 • 4 Comments

If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long for the endless immensity of the sea.

    Antoine de Saint-Exupery

Most software development projects, in fact most IT development projects have unforeseen problems that cause them to run late, overrun budgets, collapse under their own weight or get cancelled. This is best practice, it is not a problem, it only looks like a problem.

A Thought Experiment

A bank has an annual budget of $10 million for internal software projects, all projects are the same size and complexity. Over the years it has collected all sorts of data about the projects it runs, it has found that projects have fallen into two distinct groups:

Group One
These are projects run by standard development groups and behave pretty normally. They normally succeed, but often take longer and cost more than originally expected and sometimes have unforeseen problems, during or after development. In general things work out, but very few projects go completely to plan, approximately 15% of projects are cancelled outright and never result in anything usable. Projects normally cost about $100,000 to complete, involve a number of developers, testers, a documenter and a project manager. They typically take 3-6 months

Group Two
These projects are critical and use the critical development process, they are no larger nor more complicated than group one projects, but cost of error or project failure is very high (e.g. death, bankruptcy etc). These projects are very thoroughly researched, designed and developed, they are verified using tools and manual review at each stage and all products are formally tested before being integrated into the whole product. Prototypes are created to illuminate unfamiliar areas. All issues are tracked back to their point of origin, analysed and the software is checked for errors with the same pathology or method of introduction, sometimes new processes are added to the methodology to prevent similar errors being introduced. The bank has never cancelled one of these projects after initial evaluation and so far, they have all been completed to time and budget. Projects normally cost $1 million, involve several inter-dependant development teams each with its own testers, project manager and documenter. There is an independent test team that continually tests the products of the development teams, a documentation team, a project board and an overall project manager. These projects typically take 12-24 months.

In summary group one projects cost $100,000 and take 3-6 months, but are not fully controlled, 15% failure. Group two projects of the SAME SIZE and COMPLEXITY cost 10 times more and take much longer, but always work as expected.

The Software development manager moves to a new position where he is advising blue chip multinationals about introducing in house software development, none of these blue chip companies use software that could kill anyone or bankrupt the company, which process should be suggest:

        Group 1
    or Group 2


For $10 million you can have

Group 1:
10,000,000 / 100,000) * 85% and some hassle because project don’t run to plan.
=> 85 successful projects.

Group 2:
(10,000,000 / 1,000,000) * 100% and some hassle because the development process is so much work.
=> 10 successful projects.

And even though group 1 projects overrun, they still take less time than group 2 projects, because most of the overrun is unforeseen work, not mistakes.


Group 1, every time.

In the real world

The European information technology observatory( estimates total IT spending for 2007 will exceed 2 Trillion US$, that a whole lot, about one Microsoft every 8 weeks. Clearly some of this is basic office equipment and other zero risk expenses, but using almost any percentage you care to name (I seem to remember 40% from somewhere) there is still an awful lot being spent on new developments and other higher risk projects.

This is important because the numbers are large enough that statistical effects are overwhelming.

Best practice is not to work out every detail in advance, not to test everything to absolute certainty, best practice is to work on the fine detail of a project on the project. In best practice, we find things out after the project has started, in best practice projects often run late, overrun budgets and occasionally get cancelled.

Best practice is not an excuse to do a bad job, good staff, concerted effort at all stages and sound practices are still needed. However, shit happens, live with it.

For some reasons why this might be see What is obvious to me now and The very brilliant, Facts and Fallacies of Software Engineering will be useful as well.

This is best practice, it is not a problem, it only looks like a problem.


The very brilliant, Facts and Fallacies of Software Engineering

•March 26, 2007 • 1 Comment

This has to be one of the best software books I have ever read.

Why you might ask, well:

  • Its short, a mere 184 pages
  • It contains actual facts, not just opinions.
  • It contains useful actual facts.
  • It isn’t tied to any particular technology with a half life of a fruit fly (e.g. Teach Yourself Java in 21 Minutes)

So without further a do, I present the contents, a list I shall be referencing many times in the future.

55 Facts

10 reasons why software managers tend to suck.

•March 26, 2007 • 7 Comments

A developer perspective:

Most project managers, program managers, product managers, development managers are not pointy haired bosses, most are trying to do a good job, but:

    1. Software Managers are not in control.
    2. Software Managers know less than the people they are managing.
    3. Software Managers are not in control.
    4. Software Managers don’t get to do the fun stuff.
    5. Software Managers are not in control.
    6. Saying something is so, does not make it so.
    7. Software Managers are not in control.
    8. Software Managers cannot estimate how long things will take.
    9. Software Managers are not in control.
    10. We know software management is hard, because there are so few good software managers.

    1. You are not in control.

    The factor that most controls the development is not the management, it is the developers, there is a massive range in the ability of developers, it is difficult for another programmer to gauge this difference and nearly impossible for a non-programmer. Unhappy developers are unproductive, especially the good ones.

    Nearly all developers are highly motivated to produce good software, a lot of management activity negatively affects this, so a light hand is good. Almost all developers know more about the software and what they are doing with it than anyone else. (this is the details of the stuff they are buried in every day).

    The best thing that a software manager can do, is get the best developers he (sometimes she) can get hold of and remove as many obstacles as possible from them getting the job done.

    There is nothing you can do to make it work, you have to help it work.

    To many managers, getting rid of the arrogant, undisciplined, over-paid, technology-obsessed, improperly-dressed etc. programmers would appear to be a significant added benefit.

    Bjarne Stroustrup The C++ Programming Language 3e, section 24.2.4

    2. You know less than the people you are managing.

    Detailed software knowledge has a half-life of about 3 years, if you are a manager you out of date.

    When someone is working all day everyday in the details of the product, when they run the software time and again during the day, when they are thinking in real detail about how it works day in day out, they know the software. Management simply cannot apply this level of attention to every part of the product.

    Software development has chaos theory as its in-laws, you often need to know the details to make the right decision.

    3. You are not in control.

    Management and developers are motivated by different things, some managers do not understand this.

    Software developers are motivated more by problem solving, intellectual challenge, technical development, being proud of their work than they are directly by money or promotion (Nine Things Developers Want More Than Money). Many developers are positively averse to office politics.

    Think maslow’s hierarchy of needs. Think self-actualization.

    4. You don’t get to do the fun stuff.

    Every single, ex-developer I have ever spoken to misses development. Every single one. You may be better, you may be interested, you may want to keep your hand in, but you are not available, you need to focus on being a good manager. It is regrettable, but you need to live with it. Some mangers eventually go back to development, most do not.

    True happiness comes from the joy of deeds well done, the zest of creating things new.

    Antoine de Saint-Exupery

    5. You are not in control.

    You cannot really tell what developers are actually doing, if you say do it this way and they agree, how do you know what they actually do.

    If you work “with” the developers, then they will work “with” you.

    6. Saying something is so, does not make it so.

    For a lot of management you can say something is so, and proceed with the assumption that it is, even when it isn’t it is unlikely to be provable, so it can simply be asserted. It is not so in software, if the estimate for development is nine months then it is nine months. If a customer absolutely positively must have it in six months, then it won’t happen, you either have to remove a lot of functionality (perhaps half, things being non-linear) or convince the client it will take nine months.

    The most likely outcome from this clash of timetables is that the developers (and prior to that the manager) will cave in, accept the new date, rush the development which will end up taking nine months anyway (if your lucky) and be in a worse condition because its been rushed.

    7. You are not in control.

    Developers are analytical & critical thinkers, they are far more concerned with actually correct than politically correct.

    They are going to argue and discuss things and sometimes disagree with you and each other.

    The truth does not change according to our ability to stomach it.

    Flannery O’Connor

    8. You cannot estimate how long things will take.

    Estimation should be performed by the people who are going to implement the functionality. However most estimates are produced by management or marketing and typically adjusted by politics, rather than prudence. This is just lying to yourselves, don’t do it.

    Estimates are frequently determined before requirements, they are aspirations or at best guidelines. Estimates have to be based on requirements (and may require significant investigation, such as functional specifications, use cases, prototypes etc), in any case it is likely that either requirements will have to slip or the overall schedule will. You can either explain this to the customer at the beginning of a project or the end, its up to you.

    I have always found that plans are useless, but planning is indispensable.

    Dwight Eisenhower

    9. You are not in control.

    If your developers never cause you any problems, then they don’t know what they are doing. You have the wrong developers, your pretty much sunk regardless.

    You can have an easy life getting nothing done or a more “interesting” life doing something useful. You choose.

    10. We know software management is hard, because there are so few good software managers.

    In my time I have worked with the many and varied of the software world and this is how it measures:

        36% Good
        32% Average
        32% Poor

    Software Managers:
        13% Good
        23% Average
        64% Poor

    In the case of the software managers, that 13% good figure translates to three people out of nearly 15 years of experience.

    My three good managers have been:

        Mike Bates (BJSS)
        Boyd Moffat (Three-X Communications)
        Don Kavanagh (Morse)

    I know software management is hard, because there are so few that are good at it.

    If you think your management doesn’t know what it’s doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.

    Edward Yourdon, Rise and Resurrection of the American Programmer

    Add Bookmark: 


The meaning of Life

•March 25, 2007 • 4 Comments

The meaning of life  … is that it stops.

Your purpose is to make things better for having lived. Thats it.

Clarity is King

•March 11, 2007 • 5 Comments

On a clear day I can see forever.

Clarity is good.  When I look at my code, the bit I am writing right now, if it is clear, I can understand what it does, why and how.  If I look around at the other pieces of code I am using and they are clear I can understand what they are doing and why, and no doubt if I care to look inside them I can discover how, though because they are clear I do not need to do that.  It doesn’t matter if I wrote them, or someone else did, because their intention is clear.

If my project manager looks at my software and all is clear, then even though he doesn’t understand what I have been doing, certainly not in any detail, he can still understand what it does and why.

If my users look at my software and all is clear, then even though they have never seen the source code, nor programmed a line of code in their life, and even if they have, it doesn’t matter because the intention is clear and they understand how the software works and why.  They can see how to do what they want to do, without needless detail nor resorting to manuals or forums or google.

If everything is clear, everything is good.  And if everything is not good, I can fix it, because its clear what is not good and why and how.

So when I name my variables and functions and classes I think for a moment, so I can be clear about what I want this thing to do.  And if I change my mind, because something has become clear to me, then I change the name of the variable or method or class so it is clear for everyone. Sometimes a class or function or variable is not used clearly and is confused, so I change it, split it up, merge it together, rewrite it until it is clear or as clear as I can make it right now. But I come back to it.

You might think, um, doesn’t this take time, well yes it does, but you have to speculate to accumulate and I do and so I do.  So when the software is complete and everyone can see it is complete, it took less time and works better.  Though that is not always obvious to everyone.

And it is good and I know it is good, because it is clear, at least to me.

There have been studies made, many, many studies and it turns out almost all programming is thinking, of the order of 80% (Glass, Robert L.  Facts and Fallacies of Software Engineering – Fact 22 – I implore you to read this book, if you havn’t already, it contains what appear to be actual facts), in fact almost all software development regardless of the activity is thinking, of the order of 70% (Fred Brookes via Allen Holub here).  So if it is clear, I can get to understanding more quickly and from there I can get to the right answer, but if it is not clear, then I will make mistakes and invalid assumptions and do the wrong thing.  Then I have two problems, the original problem and the mistakes I have made.

In fact if everything is clear, the odds are good that I will ‘see’ a problem just by reading through the code, while I am working on it. Or maybe the code will tell me I have got it wrong, because of instrumentation I have put in it.

So I try and make everything clear, because clear begets elegance, which is better even than cuspy.

Add Bookmark: 


What is obvious to me now.

•March 4, 2007 • 9 Comments

Software IS Design:  Part 1

Corinthians I 13:11:
“When I was a child, I talked like a child, I thought like a child, I reasoned like a child. When I became a man, I put childish ways behind me.”

But I have still got so much to learn.

I don’t remember what kind of activity I originally thought programming was, maybe I hadn’t formed an idea. Engineering I think, certainly I did some software engineering at University. I still remember my lecturer, he said if ain’t tested it doesn’t work, and strive for low coupling and high cohesion. That’s about it. Good advice though.

Sometime in my C++ days I learnt from Steve McConnell that software development was definitely a construction process and I learnt a great many other things as well. He also thought low coupling and high cohesion were important and that the most important part of producing software was, well, producing the software, the programming, since it was the only piece guaranteed to be done and probably the only accurate description of the software anyway.

I held this view of building up software from its component pieces for a long time.

The problem with software viewed as construction (still an engineering task really), is it should be more predictable than it is.

  • It normally takes longer and cost more than we thought, unlike houses.
  • We can’t even work out how long it will take, unlike bridges.
  • We don’t actually know in real detail what we will actually get, how it will work and how fast it will run, unlike PCs.
  • And programmers are difficult to manage, they can’t just be replaced with an equally skilled alternative. When things get behind, adding more just seems to make things worse. Not like construction workers at all.

Then after quite a while, I came across software development as a purely design process:

  • Architectural design is design.
  • Functional or detailed design is design.
  • Even Test and Debugging is design.
  • But most importantly Coding is design.


Of course it’s not, we work out all the details in the specification and then get a coder to translate it into code for us. We only need to test it, to fix the mistakes coders make.

Well, no, not quite, not by a long way. We do have a translation process and a build process, in fact we call it a build, it takes a few minutes or perhaps longer and its performed completely automatically by the compiler and linker or ‘ant’ or ‘make’ or the IDE or something like that.

The build process takes an exact set of plans, the source code, and constructs all the pieces into a finished product. As if it was a television or a house, but faster and so cheaply it might as well be free.

I shall not try to convince you that programming is software design directly for now, but show you some of the consequences of thinking of programming as design.



Design is open ended with only a poorly defined delivery Construction is specific, with a well defined delivery and timescale
Designers are not interchangeable, especially in the middle of the process. Construction equipment and personnel are largely interchangeable at any stage of a project.
Design cannot be automated. Larger parts of construction are routinely automated
Each design will be different from each other design, though there may be similar aspects Each construction will typically be the same as dozens, hundreds even millions of others.
Designers are difficult to manage and can be unpredictable. Construction workers are much easier to manage.
There are not a lot of good designers and they are hard to find. There are many good construction workers and they are significantly easier to get
There can be orders of magnitude difference in the end result from different designers. There will be much less than an order of magnitude in productivity and quality variation in construction workers.
Design is a difficult and error prone process requiring trial and error. Construction is a well understood process, with effective quality control.
Designer love to innovate. Construction workers rarely get a chance to innovate.
A good designer can transform a product, even a company.
e.g. Jonathan Ive
A good construction worker can be relied on to do his bit.

“Learning is not compulsory. Neither is survival.”
W. Edwards Deming

Add Bookmark:  











Add to Technorati Favorites

… continued in part 2, consequences of Software as Design

What would the Dalai Lama do …

•February 27, 2007 • 1 Comment

What you need to ask yourself is;
   What would the Dalai Lama do …if he were a Navy seal.

Once upon a time there was a programmer called Sisyphus and he trudged into work and toiled all day, back and forth, back and forth. No matter where he went, people did one crazy thing after another and it really got to him, ‘cos he cared y’know. If only people would do things properly and not run around like headless chickens as soon as the project got put under any pressure. Oh and thinking things through a bit more, that would be good to.

So anyway Sisyphus is waiting for the build to complete and reading a few blogs, picking up a few tips, that kind of thing. When he comes across a couple of articles:

The Path to Happiness

The first article was about the Dalai Lama and how the Chinese totally fell out with him ‘cos he wouldn’t go work for them, and then they moved in and wouldn’t give him any space to breathe, and there was all this unpleasantness and now he can’t go home anymore.

And this is really bad ‘cos, he is the spiritual leader of his people and they need him, especially with the Chinese an’ all. But is he glum, is he downcast, no he is one of the happiest people around and very mellow, but sharp too.

And so people ask him, how is it that even when life sucks your always happy?

So he says,

In identifying one’s mental state as the prime factor in achieving happiness:

“We don’t need more money, we don’t need greater success or fame, we don’t need the perfect body or even the perfect mate – right now, at this very moment, we have a mind, which is all the basic equipment we need to achieve complete happiness.”

Human happiness and human satisfaction must ultimately come from within oneself. It is wrong to expect some final satisfaction to come from money or from a computer.

Your happiness is a product of your mind not the universe around it. Changing your mind can be difficult, but with effort it can be done. Strive sincerely and you can be happy independently of your situation.

Order out of Chaos

The second quote is from the Navy Seals mantra when training for close quarters combat (an unexpected source), somewhat different than the first.

“Fast is slow. Slow is fast.” Watch your fields of fire, pick your targets, aim centre mass, squeeze the trigger. The faster you go, the more mistakes you make. The more mistakes you make, the deeper in the shit you will be. Go slow. Try not to make mistakes. Be one with the battle yet transcend it. Float above the action, survey the scene, target immediate threats, and dispatch them with ruthless efficiency.

Basically it says that if things are going very fast, slow it down. Concentrate on the immediate issues and deal with them with ruthless efficiency in a calm and precise nature. Don’t try to do everything at once, you’ll can’t, you will fail.


If you wish to keep your head, while all around you are losing theirs, then follow the Dalai Lama and the Navy Seals: Slow down, float above the situation and move serenely through the chaos with a happy smile on your face.

See also Follow Steph – Life Quotes

Add to Technorati Favorites