If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
Most software development projects, in fact most IT development projects have unforeseen problems that cause them to run late, overrun budgets, collapse under their own weight or get cancelled. This is best practice, it is not a problem, it only looks like a problem.
A Thought Experiment
A bank has an annual budget of $10 million for internal software projects, all projects are the same size and complexity. Over the years it has collected all sorts of data about the projects it runs, it has found that projects have fallen into two distinct groups:
These are projects run by standard development groups and behave pretty normally. They normally succeed, but often take longer and cost more than originally expected and sometimes have unforeseen problems, during or after development. In general things work out, but very few projects go completely to plan, approximately 15% of projects are cancelled outright and never result in anything usable. Projects normally cost about $100,000 to complete, involve a number of developers, testers, a documenter and a project manager. They typically take 3-6 months
These projects are critical and use the critical development process, they are no larger nor more complicated than group one projects, but cost of error or project failure is very high (e.g. death, bankruptcy etc). These projects are very thoroughly researched, designed and developed, they are verified using tools and manual review at each stage and all products are formally tested before being integrated into the whole product. Prototypes are created to illuminate unfamiliar areas. All issues are tracked back to their point of origin, analysed and the software is checked for errors with the same pathology or method of introduction, sometimes new processes are added to the methodology to prevent similar errors being introduced. The bank has never cancelled one of these projects after initial evaluation and so far, they have all been completed to time and budget. Projects normally cost $1 million, involve several inter-dependant development teams each with its own testers, project manager and documenter. There is an independent test team that continually tests the products of the development teams, a documentation team, a project board and an overall project manager. These projects typically take 12-24 months.
In summary group one projects cost $100,000 and take 3-6 months, but are not fully controlled, 15% failure. Group two projects of the SAME SIZE and COMPLEXITY cost 10 times more and take much longer, but always work as expected.
The Software development manager moves to a new position where he is advising blue chip multinationals about introducing in house software development, none of these blue chip companies use software that could kill anyone or bankrupt the company, which process should be suggest:
or Group 2
For $10 million you can have
10,000,000 / 100,000) * 85% and some hassle because project don’t run to plan.
=> 85 successful projects.
(10,000,000 / 1,000,000) * 100% and some hassle because the development process is so much work.
=> 10 successful projects.
And even though group 1 projects overrun, they still take less time than group 2 projects, because most of the overrun is unforeseen work, not mistakes.
Group 1, every time.
In the real world
The European information technology observatory(http://www.eito.com/) estimates total IT spending for 2007 will exceed 2 Trillion US$, that a whole lot, about one Microsoft every 8 weeks. Clearly some of this is basic office equipment and other zero risk expenses, but using almost any percentage you care to name (I seem to remember 40% from somewhere) there is still an awful lot being spent on new developments and other higher risk projects.
This is important because the numbers are large enough that statistical effects are overwhelming.
Best practice is not to work out every detail in advance, not to test everything to absolute certainty, best practice is to work on the fine detail of a project on the project. In best practice, we find things out after the project has started, in best practice projects often run late, overrun budgets and occasionally get cancelled.
Best practice is not an excuse to do a bad job, good staff, concerted effort at all stages and sound practices are still needed. However, shit happens, live with it.
For some reasons why this might be see What is obvious to me now and The very brilliant, Facts and Fallacies of Software Engineering will be useful as well.
This is best practice, it is not a problem, it only looks like a problem.