If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
Most software development projects, in fact most IT development projects have unforeseen problems that cause them to run late, overrun budgets, collapse under their own weight or get cancelled. This is best practice, it is not a problem, it only looks like a problem.
A Thought Experiment
A bank has an annual budget of $10 million for internal software projects, all projects are the same size and complexity. Over the years it has collected all sorts of data about the projects it runs, it has found that projects have fallen into two distinct groups:
Group One
These are projects run by standard development groups and behave pretty normally. They normally succeed, but often take longer and cost more than originally expected and sometimes have unforeseen problems, during or after development. In general things work out, but very few projects go completely to plan, approximately 15% of projects are cancelled outright and never result in anything usable. Projects normally cost about $100,000 to complete, involve a number of developers, testers, a documenter and a project manager. They typically take 3-6 months
Group Two
These projects are critical and use the critical development process, they are no larger nor more complicated than group one projects, but cost of error or project failure is very high (e.g. death, bankruptcy etc). These projects are very thoroughly researched, designed and developed, they are verified using tools and manual review at each stage and all products are formally tested before being integrated into the whole product. Prototypes are created to illuminate unfamiliar areas. All issues are tracked back to their point of origin, analysed and the software is checked for errors with the same pathology or method of introduction, sometimes new processes are added to the methodology to prevent similar errors being introduced. The bank has never cancelled one of these projects after initial evaluation and so far, they have all been completed to time and budget. Projects normally cost $1 million, involve several inter-dependant development teams each with its own testers, project manager and documenter. There is an independent test team that continually tests the products of the development teams, a documentation team, a project board and an overall project manager. These projects typically take 12-24 months.
In summary group one projects cost $100,000 and take 3-6 months, but are not fully controlled, 15% failure. Group two projects of the SAME SIZE and COMPLEXITY cost 10 times more and take much longer, but always work as expected.
The Software development manager moves to a new position where he is advising blue chip multinationals about introducing in house software development, none of these blue chip companies use software that could kill anyone or bankrupt the company, which process should be suggest:
Group 1
or Group 2
Well…
For $10 million you can have
Group 1:
10,000,000 / 100,000) * 85% and some hassle because project don’t run to plan.
=> 85 successful projects.
Group 2:
(10,000,000 / 1,000,000) * 100% and some hassle because the development process is so much work.
=> 10 successful projects.
And even though group 1 projects overrun, they still take less time than group 2 projects, because most of the overrun is unforeseen work, not mistakes.
Conclusion
Group 1, every time.
In the real world
The European information technology observatory(http://www.eito.com/) estimates total IT spending for 2007 will exceed 2 Trillion US$, that a whole lot, about one Microsoft every 8 weeks. Clearly some of this is basic office equipment and other zero risk expenses, but using almost any percentage you care to name (I seem to remember 40% from somewhere) there is still an awful lot being spent on new developments and other higher risk projects.
This is important because the numbers are large enough that statistical effects are overwhelming.
Best practice is not to work out every detail in advance, not to test everything to absolute certainty, best practice is to work on the fine detail of a project on the project. In best practice, we find things out after the project has started, in best practice projects often run late, overrun budgets and occasionally get cancelled.
Best practice is not an excuse to do a bad job, good staff, concerted effort at all stages and sound practices are still needed. However, shit happens, live with it.
The factor that most controls the development is not the management, it is the developers, there is a massive range in the ability of developers, it is difficult for another programmer to gauge this difference and nearly impossible for a non-programmer. Unhappy developers are unproductive, especially the good ones.
Nearly all developers are highly motivated to produce good software, a lot of management activity negatively affects this, so a light hand is good. Almost all developers know more about the software and what they are doing with it than anyone else. (this is the details of the stuff they are buried in every day).
The best thing that a software manager can do, is get the best developers he (sometimes she) can get hold of and remove as many obstacles as possible from them getting the job done.
There is nothing you can do to make it work, you have to help it work.
To many managers, getting rid of the arrogant, undisciplined, over-paid, technology-obsessed, improperly-dressed etc. programmers would appear to be a significant added benefit.
2. You know less than the people you are managing.
Detailed software knowledge has a half-life of about 3 years, if you are a manager you out of date.
When someone is working all day everyday in the details of the product, when they run the software time and again during the day, when they are thinking in real detail about how it works day in day out, they know the software. Management simply cannot apply this level of attention to every part of the product.
Software development has chaos theory as its in-laws, you often need to know the details to make the right decision.
3. You are not in control.
Management and developers are motivated by different things, some managers do not understand this.
Software developers are motivated more by problem solving, intellectual challenge, technical development, being proud of their work than they are directly by money or promotion (Nine Things Developers Want More Than Money). Many developers are positively averse to office politics.
Every single, ex-developer I have ever spoken to misses development. Every single one. You may be better, you may be interested, you may want to keep your hand in, but you are not available, you need to focus on being a good manager. It is regrettable, but you need to live with it. Some mangers eventually go back to development, most do not.
True happiness comes from the joy of deeds well done, the zest of creating things new.
You cannot really tell what developers are actually doing, if you say do it this way and they agree, how do you know what they actually do.
If you work “with” the developers, then they will work “with” you.
6. Saying something is so, does not make it so.
For a lot of management you can say something is so, and proceed with the assumption that it is, even when it isn’t it is unlikely to be provable, so it can simply be asserted. It is not so in software, if the estimate for development is nine months then it is nine months. If a customer absolutely positively must have it in six months, then it won’t happen, you either have to remove a lot of functionality (perhaps half, things being non-linear) or convince the client it will take nine months.
The most likely outcome from this clash of timetables is that the developers (and prior to that the manager) will cave in, accept the new date, rush the development which will end up taking nine months anyway (if your lucky) and be in a worse condition because its been rushed.
7. You are not in control.
Developers are analytical & critical thinkers, they are far more concerned with actually correct than politically correct.
They are going to argue and discuss things and sometimes disagree with you and each other.
The truth does not change according to our ability to stomach it.
Estimation should be performed by the people who are going to implement the functionality. However most estimates are produced by management or marketing and typically adjusted by politics, rather than prudence. This is just lying to yourselves, don’t do it.
Estimates are frequently determined before requirements, they are aspirations or at best guidelines. Estimates have to be based on requirements (and may require significant investigation, such as functional specifications, use cases, prototypes etc), in any case it is likely that either requirements will have to slip or the overall schedule will. You can either explain this to the customer at the beginning of a project or the end, its up to you.
I have always found that plans are useless, but planning is indispensable.
If your developers never cause you any problems, then they don’t know what they are doing. You have the wrong developers, your pretty much sunk regardless.
You can have an easy life getting nothing done or a more “interesting” life doing something useful. You choose.
10. We know software management is hard, because there are so few good software managers.
In my time I have worked with the many and varied of the software world and this is how it measures:
Developers:
36% Good
32% Average
32% Poor
Software Managers:
13% Good
23% Average
64% Poor
In the case of the software managers, that 13% good figure translates to three people out of nearly 15 years of experience.
My three good managers have been:
Mike Bates (BJSS)
Boyd Moffat (Three-X Communications)
Don Kavanagh (Morse)
I know software management is hard, because there are so few that are good at it.
If you think your management doesn’t know what it’s doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.
Clarity is good. When I look at my code, the bit I am writing right now, if it is clear, I can understand what it does, why and how. If I look around at the other pieces of code I am using and they are clear I can understand what they are doing and why, and no doubt if I care to look inside them I can discover how, though because they are clear I do not need to do that. It doesn’t matter if I wrote them, or someone else did, because their intention is clear.
If my project manager looks at my software and all is clear, then even though he doesn’t understand what I have been doing, certainly not in any detail, he can still understand what it does and why.
If my users look at my software and all is clear, then even though they have never seen the source code, nor programmed a line of code in their life, and even if they have, it doesn’t matter because the intention is clear and they understand how the software works and why. They can see how to do what they want to do, without needless detail nor resorting to manuals or forums or google.
If everything is clear, everything is good. And if everything is not good, I can fix it, because its clear what is not good and why and how.
So when I name my variables and functions and classes I think for a moment, so I can be clear about what I want this thing to do. And if I change my mind, because something has become clear to me, then I change the name of the variable or method or class so it is clear for everyone. Sometimes a class or function or variable is not used clearly and is confused, so I change it, split it up, merge it together, rewrite it until it is clear or as clear as I can make it right now. But I come back to it.
You might think, um, doesn’t this take time, well yes it does, but you have to speculate to accumulate and I do and so I do. So when the software is complete and everyone can see it is complete, it took less time and works better. Though that is not always obvious to everyone.
And it is good and I know it is good, because it is clear, at least to me.
There have been studies made, many, many studies and it turns out almost all programming is thinking, of the order of 80% (Glass, Robert L. Facts and Fallacies of Software Engineering – Fact 22 – I implore you to read this book, if you havn’t already, it contains what appear to be actual facts), in fact almost all software development regardless of the activity is thinking, of the order of 70% (Fred Brookes via Allen Holub here). So if it is clear, I can get to understanding more quickly and from there I can get to the right answer, but if it is not clear, then I will make mistakes and invalid assumptions and do the wrong thing. Then I have two problems, the original problem and the mistakes I have made.
In fact if everything is clear, the odds are good that I will ‘see’ a problem just by reading through the code, while I am working on it. Or maybe the code will tell me I have got it wrong, because of instrumentation I have put in it.
So I try and make everything clear, because clear begets elegance, which is better even than cuspy.
Corinthians I 13:11:
“When I was a child, I talked like a child, I thought like a child, I reasoned like a child. When I became a man, I put childish ways behind me.”
But I have still got so much to learn.
I don’t remember what kind of activity I originally thought programming was, maybe I hadn’t formed an idea. Engineering I think, certainly I did some software engineering at University. I still remember my lecturer, he said if ain’t tested it doesn’t work, and strive for low coupling and high cohesion. That’s about it. Good advice though.
Sometime in my C++ days I learnt from Steve McConnell that software development was definitely a construction process and I learnt a great many other things as well. He also thought low coupling and high cohesion were important and that the most important part of producing software was, well, producing the software, the programming, since it was the only piece guaranteed to be done and probably the only accurate description of the software anyway.
I held this view of building up software from its component pieces for a long time.
The problem with software viewed as construction (still an engineering task really), is it should be more predictable than it is.
It normally takes longer and cost more than we thought, unlike houses.
We can’t even work out how long it will take, unlike bridges.
We don’t actually know in real detail what we will actually get, how it will work and how fast it will run, unlike PCs.
And programmers are difficult to manage, they can’t just be replaced with an equally skilled alternative. When things get behind, adding more just seems to make things worse. Not like construction workers at all.
Then after quite a while, I came across software development as a purely design process:
Architectural design is design.
Functional or detailed design is design.
Even Test and Debugging is design.
But most importantly Coding is design.
What!
Of course it’s not, we work out all the details in the specification and then get a coder to translate it into code for us. We only need to test it, to fix the mistakes coders make.
Well, no, not quite, not by a long way. We do have a translation process and a build process, in fact we call it a build, it takes a few minutes or perhaps longer and its performed completely automatically by the compiler and linker or ‘ant’ or ‘make’ or the IDE or something like that.
The build process takes an exact set of plans, the source code, and constructs all the pieces into a finished product. As if it was a television or a house, but faster and so cheaply it might as well be free.
I shall not try to convince you that programming is software design directly for now, but show you some of the consequences of thinking of programming as design.
Design
Construction
Design is open ended with only a poorly defined delivery
Construction is specific, with a well defined delivery and timescale
Designers are not interchangeable, especially in the middle of the process.
Construction equipment and personnel are largely interchangeable at any stage of a project.
Design cannot be automated.
Larger parts of construction are routinely automated
Each design will be different from each other design, though there may be similar aspects
Each construction will typically be the same as dozens, hundreds even millions of others.
Designers are difficult to manage and can be unpredictable.
Construction workers are much easier to manage.
There are not a lot of good designers and they are hard to find.
There are many good construction workers and they are significantly easier to get
There can be orders of magnitude difference in the end result from different designers.
There will be much less than an order of magnitude in productivity and quality variation in construction workers.
Design is a difficult and error prone process requiring trial and error.
Construction is a well understood process, with effective quality control.
Designer love to innovate.
Construction workers rarely get a chance to innovate.
A good designer can transform a product, even a company.
e.g. Jonathan Ive
A good construction worker can be relied on to do his bit.
“Learning is not compulsory. Neither is survival.” W. Edwards Deming
Add Bookmark:
… continued in part 2, consequences of Software as Design
What you need to ask yourself is;
What would the Dalai Lama do …if he were a Navy seal.
Once upon a time there was a programmer called Sisyphus and he trudged into work and toiled all day, back and forth, back and forth. No matter where he went, people did one crazy thing after another and it really got to him, ‘cos he cared y’know. If only people would do things properly and not run around like headless chickens as soon as the project got put under any pressure. Oh and thinking things through a bit more, that would be good to.
So anyway Sisyphus is waiting for the build to complete and reading a few blogs, picking up a few tips, that kind of thing. When he comes across a couple of articles:
The first article was about the Dalai Lama and how the Chinese totally fell out with him ‘cos he wouldn’t go work for them, and then they moved in and wouldn’t give him any space to breathe, and there was all this unpleasantness and now he can’t go home anymore.
And this is really bad ‘cos, he is the spiritual leader of his people and they need him, especially with the Chinese an’ all. But is he glum, is he downcast, no he is one of the happiest people around and very mellow, but sharp too.
And so people ask him, how is it that even when life sucks your always happy?
So he says,
In identifying one’s mental state as the prime factor in achieving happiness:
“We don’t need more money, we don’t need greater success or fame, we don’t need the perfect body or even the perfect mate – right now, at this very moment, we have a mind, which is all the basic equipment we need to achieve complete happiness.”
Human happiness and human satisfaction must ultimately come from within oneself. It is wrong to expect some final satisfaction to come from money or from a computer.
Your happiness is a product of your mind not the universe around it. Changing your mind can be difficult, but with effort it can be done. Strive sincerely and you can be happy independently of your situation.
Order out of Chaos
The second quote is from the Navy Seals mantra when training for close quarters combat (an unexpected source), somewhat different than the first.
“Fast is slow. Slow is fast.” Watch your fields of fire, pick your targets, aim centre mass, squeeze the trigger. The faster you go, the more mistakes you make. The more mistakes you make, the deeper in the shit you will be. Go slow. Try not to make mistakes. Be one with the battle yet transcend it. Float above the action, survey the scene, target immediate threats, and dispatch them with ruthless efficiency.
Basically it says that if things are going very fast, slow it down. Concentrate on the immediate issues and deal with them with ruthless efficiency in a calm and precise nature. Don’t try to do everything at once, you’ll can’t, you will fail.
So…
If you wish to keep your head, while all around you are losing theirs, then follow the Dalai Lama and the Navy Seals: Slow down, float above the situation and move serenely through the chaos with a happy smile on your face.
A long time ago, Ptolemy and Aristotle told us the earth is the centre of the universe and that the sun, the planets and the moon all orbit the earth, and the stars are fixed to the firmament.
People observed the motion of the sun and the moon and this was ok, but when they observed the planets they behaved in very strange ways, first moving one way, then moving back, then moving forward again. Theories were put forward to explain this motion. But eventually having decided that God would not make an imperfect universe Copernicus and later Galileo told us that the earth and planets orbited the sun and only the moon orbited the earth. Here was a simple solution that didn’t require lots of special cases, it was consistent and easy to understand in fact it was elegant. It was also correct.
It was Galileo who said
“Why is it that the objects of mathematical creation, satisfying the criteria of beauty and simplicity, are precisely the ones that time and time again are found to be essential for a correct description of the external world?”
And not just Galileo, physicists know the universe is elegant, the great theoretical physicist, Paul A.M. Dirac, said in his 1963 article in Scientific American:
“A theory with mathematical beauty is more likely to be correct than an ugly one that fits some experimental data. God is a mathematician of a very high order.”
And so it is with software.
When you start a project, at first you create a small thing, then using this small thing you test it, add pieces and test it again, until its finished.
From day one, the earlier work has become the tools and building blocks of the later work. If even a little time is spent on elegance then all is consistent, flexible, easy to change and a delight to work with, the project flows and success is assured.
But if no thought is spared for elegance, then the project becomes a battle, execution is impaired, understanding lost, progress is slowed; change is difficult and failure looms.
Elegance makes software consistent and predictable, it reduces complexity more and more as time goes by and above all it makes the software understandable. In short, if you want to build stuff that works and lasts quickly, seek out elegance in every corner of your design, it will pay dividends again and again and again.
I can't believe it's not better 