The real reason so many software projects are late, over budget and cancelled.

If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long for the endless immensity of the sea.

    Antoine de Saint-Exupery

Most software development projects, in fact most IT development projects have unforeseen problems that cause them to run late, overrun budgets, collapse under their own weight or get cancelled. This is best practice, it is not a problem, it only looks like a problem.

A Thought Experiment

A bank has an annual budget of $10 million for internal software projects, all projects are the same size and complexity. Over the years it has collected all sorts of data about the projects it runs, it has found that projects have fallen into two distinct groups:

Group One
These are projects run by standard development groups and behave pretty normally. They normally succeed, but often take longer and cost more than originally expected and sometimes have unforeseen problems, during or after development. In general things work out, but very few projects go completely to plan, approximately 15% of projects are cancelled outright and never result in anything usable. Projects normally cost about $100,000 to complete, involve a number of developers, testers, a documenter and a project manager. They typically take 3-6 months

Group Two
These projects are critical and use the critical development process, they are no larger nor more complicated than group one projects, but cost of error or project failure is very high (e.g. death, bankruptcy etc). These projects are very thoroughly researched, designed and developed, they are verified using tools and manual review at each stage and all products are formally tested before being integrated into the whole product. Prototypes are created to illuminate unfamiliar areas. All issues are tracked back to their point of origin, analysed and the software is checked for errors with the same pathology or method of introduction, sometimes new processes are added to the methodology to prevent similar errors being introduced. The bank has never cancelled one of these projects after initial evaluation and so far, they have all been completed to time and budget. Projects normally cost $1 million, involve several inter-dependant development teams each with its own testers, project manager and documenter. There is an independent test team that continually tests the products of the development teams, a documentation team, a project board and an overall project manager. These projects typically take 12-24 months.

In summary group one projects cost $100,000 and take 3-6 months, but are not fully controlled, 15% failure. Group two projects of the SAME SIZE and COMPLEXITY cost 10 times more and take much longer, but always work as expected.

The Software development manager moves to a new position where he is advising blue chip multinationals about introducing in house software development, none of these blue chip companies use software that could kill anyone or bankrupt the company, which process should be suggest:

        Group 1
    or Group 2

Well…

For $10 million you can have

Group 1:
10,000,000 / 100,000) * 85% and some hassle because project don’t run to plan.
=> 85 successful projects.

Group 2:
(10,000,000 / 1,000,000) * 100% and some hassle because the development process is so much work.
=> 10 successful projects.

And even though group 1 projects overrun, they still take less time than group 2 projects, because most of the overrun is unforeseen work, not mistakes.

Conclusion

Group 1, every time.

In the real world

The European information technology observatory(http://www.eito.com/) estimates total IT spending for 2007 will exceed 2 Trillion US$, that a whole lot, about one Microsoft every 8 weeks. Clearly some of this is basic office equipment and other zero risk expenses, but using almost any percentage you care to name (I seem to remember 40% from somewhere) there is still an awful lot being spent on new developments and other higher risk projects.

This is important because the numbers are large enough that statistical effects are overwhelming.

Best practice is not to work out every detail in advance, not to test everything to absolute certainty, best practice is to work on the fine detail of a project on the project. In best practice, we find things out after the project has started, in best practice projects often run late, overrun budgets and occasionally get cancelled.

Best practice is not an excuse to do a bad job, good staff, concerted effort at all stages and sound practices are still needed. However, shit happens, live with it.

For some reasons why this might be see What is obvious to me now and The very brilliant, Facts and Fallacies of Software Engineering will be useful as well.

This is best practice, it is not a problem, it only looks like a problem.

The very brilliant, Facts and Fallacies of Software Engineering

This has to be one of the best software books I have ever read.

Why you might ask, well:

• Its short, a mere 184 pages
• It contains actual facts, not just opinions.
• It contains useful actual facts.
• It isn’t tied to any particular technology with a half life of a fruit fly (e.g. Teach Yourself Java in 21 Minutes)

So without further a do, I present the contents, a list I shall be referencing many times in the future.

55 Facts
ABOUT MANAGEMENT
PEOPLE
Fact 1	The most important factor in software work is the quality of the programmers.
Fact 2	The best programmers are up to 28 times better than the worst programmers.
Fact 3	Adding people to a late project makes it later.
Fact 4	The working environment has a profound impact on productivity and quality.
TOOLS AND TECHNIQUES
Fact 5	Hype (about tools and techniques) is the plague on the house of software.
Fact 6	New tools and techniques cause an initial loss of productivity/quality.
Fact 7	Software developers talk a lot about tools, but seldom use them.
ESTIMATION
Fact 8	One of the two most common causes of runaway projects is poor estimation.
Fact 9	Software estimation usually occurs at the wrong time.
Fact 10	Software estimation is usually done by the wrong people.
Fact 11	Software estimates are rarely corrected as the project proceeds.
Fact 12	It is not surprising that software estimates are bad. But we live and die by them anyway!
Fact 13	There is a disconnect between software management and their programmers.
Fact 14	The answer to a feasability study is almost always “yes.”
REUSE
Fact 15	Reuse-in-the-small is a well-solved problem.
Fact 16	Reuse-in-the-large remains a mostly-unsolved problem.
Fact 17	Reuse-in-the-large works best for families of related systems.
Fact 18	Reusable components are three times as hard to build and should be tried out in three settings.
Fact 19	Modification of reused code is particularly error-prone.
Fact 20	Design pattern reuse is one solution to the problem of code reuse.
COMPLEXITY
Fact 21	For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity.
Fact 22	Eighty percent of software work is intellectual. A fair amount of it is creative. Little of it is clerical.
ABOUT THE LIFE CYCLE
REQUIREMENTS
Fact 23	One of the two most common causes of runaway projects is unstable requirements.
Fact 24	Requirements errors are the most expensive to fix during production.
Fact 25	Missing requirements are the hardest requirements errors to correct.
DESIGN
Fact 26	Explicit requirements “explode” as implicit (design) requirements for a solition evolve.
Fact 27	There is seldom one best design solution to a software problem.
Fact 28	Design is a complex, iterative process. Initial design solutions are usually wrong and certainly not optimal.
CODING
Fact 29	Designer “primitives (solutions programmers can readily code) rarely match programmer “primitives.”
Fact 30	COBOL is a very bad language, but all the others (for business applications) are so much worse.
ERROR REMOVAL
Fact 31	Error removal is the most time-consuming phase of the life cycle.
TESTING
Fact 32	Software is usually tested at best 55 to 60 precent (branch) coverage level.
Fact 33	One hundred percent coverage is still far from enough.
Fact 34	Test tools are essential, but many are rarely used.
Fact 35	Test automation rarely is. Most testing activities cannot be automated.
Fact 36	Programmer-created, built-in debug code is an important supplement to testing tools.
REVIEWS AND INSPECTIONS
Fact 37	Rigorous inspections can remove up to 90 percent of errors before the first test case is run.
Fact 38	Rigorous inspections should not replace testing.
Fact 39	Postdelivery reviews (some call them “retrospectives”) are important and seldon perfromed.
Fact 40	Reviews are both technical and socialogical, and both factors must be accommodated.
MAINTENANCE
Fact 41	Maintenance typically consumes 50 to 80 percent of software costs. It is probably the most important life cycle phase of software.
Fact 42	Enhancements represent roughly 60 percent of maintenance costs.
Fact 43	Maintenance is a solution, not a problem.
Fact 44	Understanding the existing product is the most difficult task of maintenance.
Fact 45	Better methods lead to more maintenance, not less.
ABOUT QUALITY
QUALITY
Fact 46	Quality is a collection of attributes.
Fact 47	Quality is not user satisfaction, meeting requirements, achieveing cost and schedule, or reliability.
RELIABILITY
Fact 48	There are errors that most programmers tend to make.
Fact 49	Errors tend to cluster.
Fact 50	There is no single best approach to software error removal.
Fact 51	Residual errors will always persist. The goal should be to minimize or eliminate severe errors.
EFFICIENCY
Fact 52	Efficiency stems more from good design than from good coding.
Fact 53	High-order language code can be about 90 percent as efficient as comparable assembler code.
Fact 54	There are tradeoffs between size and time optimization.
ABOUT RESEARCH
Fact 55	Many researchers advocate rather than investigate.
5+5 Fallacies
ABOUT MANAGEMENT
Fallacy 1	You can’t manage what you can’t measure.
Fallacy 2	You can manage quality into a software product.
PEOPLE
Fallacy 3	Programming can and should be egoless.
TOOLS AND TECHNIQUES
Fallacy 4	Tools and tecniques: one size fits all.
Fallacy 5	Software needs more methodologies.
ESTIMATION
Fallacy 6	To estimate cost and schedule, first estimate lines of code.
ABOUT THE LIFE CYCLE
TESTING
Fallacy 7	Random test input is a good way to optimize testing.
REVIEWS
Fallacy 8	“Given enough eyeballs, all bugs are shallow.”
MAINTENANCE
Fallacy 9	The way to predict the future maintenance costs and to make product replacement decisions is to look at past cost data.
ABOUT EDUCATION
Fallacy 10	You can teach people how to program by showing them how to write programs.

10 reasons why software managers tend to suck.

A developer perspective:

Most project managers, program managers, product managers, development managers are not pointy haired bosses, most are trying to do a good job, but:

1. Software Managers are not in control.
2. Software Managers know less than the people they are managing.
3. Software Managers are not in control.
4. Software Managers don’t get to do the fun stuff.
5. Software Managers are not in control.
6. Saying something is so, does not make it so.
7. Software Managers are not in control.
8. Software Managers cannot estimate how long things will take.
9. Software Managers are not in control.
10. We know software management is hard, because there are so few good software managers.

1. You are not in control.

• The most important factor in developing software is the quality of the programmers.
• The best programmers are up to 28 times better than the worst programmers (in quality as well as productivity)
• The working environment has a profound impact on productivity and quality.

The factor that most controls the development is not the management, it is the developers, there is a massive range in the ability of developers, it is difficult for another programmer to gauge this difference and nearly impossible for a non-programmer. Unhappy developers are unproductive, especially the good ones.

Nearly all developers are highly motivated to produce good software, a lot of management activity negatively affects this, so a light hand is good. Almost all developers know more about the software and what they are doing with it than anyone else. (this is the details of the stuff they are buried in every day).

The best thing that a software manager can do, is get the best developers he (sometimes she) can get hold of and remove as many obstacles as possible from them getting the job done.

There is nothing you can do to make it work, you have to help it work.

To many managers, getting rid of the arrogant, undisciplined, over-paid, technology-obsessed, improperly-dressed etc. programmers would appear to be a significant added benefit.

Bjarne Stroustrup The C++ Programming Language 3e, section 24.2.4

2. You know less than the people you are managing.

Detailed software knowledge has a half-life of about 3 years, if you are a manager you out of date.

When someone is working all day everyday in the details of the product, when they run the software time and again during the day, when they are thinking in real detail about how it works day in day out, they know the software. Management simply cannot apply this level of attention to every part of the product.

Software development has chaos theory as its in-laws, you often need to know the details to make the right decision.

3. You are not in control.

Management and developers are motivated by different things, some managers do not understand this.

Software developers are motivated more by problem solving, intellectual challenge, technical development, being proud of their work than they are directly by money or promotion (Nine Things Developers Want More Than Money). Many developers are positively averse to office politics.

Think maslow’s hierarchy of needs. Think self-actualization.



4. You don’t get to do the fun stuff.

Every single, ex-developer I have ever spoken to misses development. Every single one. You may be better, you may be interested, you may want to keep your hand in, but you are not available, you need to focus on being a good manager. It is regrettable, but you need to live with it. Some mangers eventually go back to development, most do not.

True happiness comes from the joy of deeds well done, the zest of creating things new.

Antoine de Saint-Exupery

5. You are not in control.

You cannot really tell what developers are actually doing, if you say do it this way and they agree, how do you know what they actually do.

If you work “with” the developers, then they will work “with” you.

6. Saying something is so, does not make it so.

For a lot of management you can say something is so, and proceed with the assumption that it is, even when it isn’t it is unlikely to be provable, so it can simply be asserted. It is not so in software, if the estimate for development is nine months then it is nine months. If a customer absolutely positively must have it in six months, then it won’t happen, you either have to remove a lot of functionality (perhaps half, things being non-linear) or convince the client it will take nine months.

The most likely outcome from this clash of timetables is that the developers (and prior to that the manager) will cave in, accept the new date, rush the development which will end up taking nine months anyway (if your lucky) and be in a worse condition because its been rushed.

7. You are not in control.

Developers are analytical & critical thinkers, they are far more concerned with actually correct than politically correct.

They are going to argue and discuss things and sometimes disagree with you and each other.

The truth does not change according to our ability to stomach it.

Flannery O’Connor

8. You cannot estimate how long things will take.

• Estimation is usually done by the wrong people.
• Estimation usually occurs at the wrong time.
• Software estimates are rarely corrected as the project proceeds.
• One of the two most common causes of runaway projects is poor estimation.

Estimation should be performed by the people who are going to implement the functionality. However most estimates are produced by management or marketing and typically adjusted by politics, rather than prudence. This is just lying to yourselves, don’t do it.

Estimates are frequently determined before requirements, they are aspirations or at best guidelines. Estimates have to be based on requirements (and may require significant investigation, such as functional specifications, use cases, prototypes etc), in any case it is likely that either requirements will have to slip or the overall schedule will. You can either explain this to the customer at the beginning of a project or the end, its up to you.

I have always found that plans are useless, but planning is indispensable.

Dwight Eisenhower

9. You are not in control.

If your developers never cause you any problems, then they don’t know what they are doing. You have the wrong developers, your pretty much sunk regardless.

You can have an easy life getting nothing done or a more “interesting” life doing something useful. You choose.

10. We know software management is hard, because there are so few good software managers.

In my time I have worked with the many and varied of the software world and this is how it measures:

Developers:
    36% Good
    32% Average
    32% Poor

Software Managers:
    13% Good
    23% Average
    64% Poor

In the case of the software managers, that 13% good figure translates to three people out of nearly 15 years of experience.

My three good managers have been:

    Mike Bates (BJSS)
    Boyd Moffat (Three-X Communications)
    Don Kavanagh (Morse)

I know software management is hard, because there are so few that are good at it.

---

If you think your management doesn’t know what it’s doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.

Edward Yourdon, Rise and Resurrection of the American Programmer

Add Bookmark:

The meaning of Life

The meaning of life  … is that it stops.

Your purpose is to make things better for having lived. Thats it.

Clarity is King

On a clear day I can see forever.

Clarity is good.  When I look at my code, the bit I am writing right now, if it is clear, I can understand what it does, why and how.  If I look around at the other pieces of code I am using and they are clear I can understand what they are doing and why, and no doubt if I care to look inside them I can discover how, though because they are clear I do not need to do that.  It doesn’t matter if I wrote them, or someone else did, because their intention is clear.

If my project manager looks at my software and all is clear, then even though he doesn’t understand what I have been doing, certainly not in any detail, he can still understand what it does and why.

If my users look at my software and all is clear, then even though they have never seen the source code, nor programmed a line of code in their life, and even if they have, it doesn’t matter because the intention is clear and they understand how the software works and why.  They can see how to do what they want to do, without needless detail nor resorting to manuals or forums or google.

If everything is clear, everything is good.  And if everything is not good, I can fix it, because its clear what is not good and why and how.

So when I name my variables and functions and classes I think for a moment, so I can be clear about what I want this thing to do.  And if I change my mind, because something has become clear to me, then I change the name of the variable or method or class so it is clear for everyone. Sometimes a class or function or variable is not used clearly and is confused, so I change it, split it up, merge it together, rewrite it until it is clear or as clear as I can make it right now. But I come back to it.

You might think, um, doesn’t this take time, well yes it does, but you have to speculate to accumulate and I do and so I do.  So when the software is complete and everyone can see it is complete, it took less time and works better.  Though that is not always obvious to everyone.

And it is good and I know it is good, because it is clear, at least to me.

There have been studies made, many, many studies and it turns out almost all programming is thinking, of the order of 80% (Glass, Robert L.  Facts and Fallacies of Software Engineering – Fact 22 – I implore you to read this book, if you havn’t already, it contains what appear to be actual facts), in fact almost all software development regardless of the activity is thinking, of the order of 70% (Fred Brookes via Allen Holub here).  So if it is clear, I can get to understanding more quickly and from there I can get to the right answer, but if it is not clear, then I will make mistakes and invalid assumptions and do the wrong thing.  Then I have two problems, the original problem and the mistakes I have made.

In fact if everything is clear, the odds are good that I will ‘see’ a problem just by reading through the code, while I am working on it. Or maybe the code will tell me I have got it wrong, because of instrumentation I have put in it.

So I try and make everything clear, because clear begets elegance, which is better even than cuspy.

Add Bookmark:

What is obvious to me now.

Software IS Design:  Part 1

Corinthians I 13:11:
“When I was a child, I talked like a child, I thought like a child, I reasoned like a child. When I became a man, I put childish ways behind me.”

But I have still got so much to learn.

I don’t remember what kind of activity I originally thought programming was, maybe I hadn’t formed an idea. Engineering I think, certainly I did some software engineering at University. I still remember my lecturer, he said if ain’t tested it doesn’t work, and strive for low coupling and high cohesion. That’s about it. Good advice though.

Sometime in my C++ days I learnt from Steve McConnell that software development was definitely a construction process and I learnt a great many other things as well. He also thought low coupling and high cohesion were important and that the most important part of producing software was, well, producing the software, the programming, since it was the only piece guaranteed to be done and probably the only accurate description of the software anyway.

I held this view of building up software from its component pieces for a long time.

The problem with software viewed as construction (still an engineering task really), is it should be more predictable than it is.

• It normally takes longer and cost more than we thought, unlike houses.
• We can’t even work out how long it will take, unlike bridges.
• We don’t actually know in real detail what we will actually get, how it will work and how fast it will run, unlike PCs.
• And programmers are difficult to manage, they can’t just be replaced with an equally skilled alternative. When things get behind, adding more just seems to make things worse. Not like construction workers at all.

Then after quite a while, I came across software development as a purely design process:

• Architectural design is design.
• Functional or detailed design is design.
• Even Test and Debugging is design.
• But most importantly Coding is design.

What!

Of course it’s not, we work out all the details in the specification and then get a coder to translate it into code for us. We only need to test it, to fix the mistakes coders make.

Well, no, not quite, not by a long way. We do have a translation process and a build process, in fact we call it a build, it takes a few minutes or perhaps longer and its performed completely automatically by the compiler and linker or ‘ant’ or ‘make’ or the IDE or something like that.

The build process takes an exact set of plans, the source code, and constructs all the pieces into a finished product. As if it was a television or a house, but faster and so cheaply it might as well be free.

I shall not try to convince you that programming is software design directly for now, but show you some of the consequences of thinking of programming as design.

Design	Construction
Design is open ended with only a poorly defined delivery	Construction is specific, with a well defined delivery and timescale
Designers are not interchangeable, especially in the middle of the process.	Construction equipment and personnel are largely interchangeable at any stage of a project.
Design cannot be automated.	Larger parts of construction are routinely automated
Each design will be different from each other design, though there may be similar aspects	Each construction will typically be the same as dozens, hundreds even millions of others.
Designers are difficult to manage and can be unpredictable.	Construction workers are much easier to manage.
There are not a lot of good designers and they are hard to find.	There are many good construction workers and they are significantly easier to get
There can be orders of magnitude difference in the end result from different designers.	There will be much less than an order of magnitude in productivity and quality variation in construction workers.
Design is a difficult and error prone process requiring trial and error.	Construction is a well understood process, with effective quality control.
Designer love to innovate.	Construction workers rarely get a chance to innovate.
A good designer can transform a product, even a company. e.g. Jonathan Ive	A good construction worker can be relied on to do his bit.

“Learning is not compulsory. Neither is survival.”
W. Edwards Deming

Add Bookmark:

… continued in part 2, consequences of Software as Design

What would the Dalai Lama do …

What you need to ask yourself is;
   What would the Dalai Lama do …if he were a Navy seal.

Once upon a time there was a programmer called Sisyphus and he trudged into work and toiled all day, back and forth, back and forth. No matter where he went, people did one crazy thing after another and it really got to him, ‘cos he cared y’know. If only people would do things properly and not run around like headless chickens as soon as the project got put under any pressure. Oh and thinking things through a bit more, that would be good to.

So anyway Sisyphus is waiting for the build to complete and reading a few blogs, picking up a few tips, that kind of thing. When he comes across a couple of articles:

The Path to Happiness

The first article was about the Dalai Lama and how the Chinese totally fell out with him ‘cos he wouldn’t go work for them, and then they moved in and wouldn’t give him any space to breathe, and there was all this unpleasantness and now he can’t go home anymore.

And this is really bad ‘cos, he is the spiritual leader of his people and they need him, especially with the Chinese an’ all. But is he glum, is he downcast, no he is one of the happiest people around and very mellow, but sharp too.

And so people ask him, how is it that even when life sucks your always happy?

So he says,

In identifying one’s mental state as the prime factor in achieving happiness:

“We don’t need more money, we don’t need greater success or fame, we don’t need the perfect body or even the perfect mate – right now, at this very moment, we have a mind, which is all the basic equipment we need to achieve complete happiness.”

Human happiness and human satisfaction must ultimately come from within oneself. It is wrong to expect some final satisfaction to come from money or from a computer.

Your happiness is a product of your mind not the universe around it. Changing your mind can be difficult, but with effort it can be done. Strive sincerely and you can be happy independently of your situation.

Order out of Chaos

The second quote is from the Navy Seals mantra when training for close quarters combat (an unexpected source), somewhat different than the first.

“Fast is slow. Slow is fast.” Watch your fields of fire, pick your targets, aim centre mass, squeeze the trigger. The faster you go, the more mistakes you make. The more mistakes you make, the deeper in the shit you will be. Go slow. Try not to make mistakes. Be one with the battle yet transcend it. Float above the action, survey the scene, target immediate threats, and dispatch them with ruthless efficiency.

Basically it says that if things are going very fast, slow it down. Concentrate on the immediate issues and deal with them with ruthless efficiency in a calm and precise nature. Don’t try to do everything at once, you’ll can’t, you will fail.

So…

If you wish to keep your head, while all around you are losing theirs, then follow the Dalai Lama and the Navy Seals: Slow down, float above the situation and move serenely through the chaos with a happy smile on your face.

See also Follow Steph – Life Quotes

Why Elegance.

Elegance makes the universe work.  

A long time ago, Ptolemy and Aristotle told us the earth is the centre of the universe and that the sun, the planets and the moon all orbit the earth, and the stars are fixed to the firmament. 

People observed the motion of the sun and the moon and this was ok, but when they observed the planets they behaved in very strange ways, first moving one way, then moving back, then moving forward again.  Theories were put forward to explain this motion. But eventually having decided that God would not make an imperfect universe Copernicus and later Galileo told us that the earth and planets orbited the sun and only the moon orbited the earth.  Here was a simple solution that didn’t require lots of special cases, it was consistent and easy to understand in fact it was elegant.  It was also correct.  

  It was Galileo who said

“Why is it that the objects of mathematical creation, satisfying the criteria of beauty and simplicity, are precisely the ones that time and time again are found to be essential for a correct description of the external world?” 

And not just Galileo, physicists know the universe is elegant, the great theoretical physicist, Paul A.M. Dirac, said in his 1963 article in Scientific American:

“A theory with mathematical beauty is more likely to be correct than an ugly one that fits some experimental data. God is a mathematician of a very high order.”  

And so it is with software.  

When you start a project, at first you create a small thing, then using this small thing you test it, add pieces and test it again, until its finished.

From day one, the earlier work has become the tools and building blocks of the later work.  If even a little time is spent on elegance then all is consistent, flexible, easy to change and a delight to work with, the project flows and success is assured.

But if no thought is spared for elegance, then the project becomes a battle, execution is impaired, understanding lost, progress is slowed; change is difficult and failure looms.  

Elegance makes software consistent and predictable, it reduces complexity more and more as time goes by and above all it makes the software understandable. In short, if you want to build stuff that works and lasts quickly, seek out elegance in every corner of your design, it will pay dividends again and again and again.