Risk Management Lessons Learned from FBI Debacle

Major IT projects tend to present many small problems, and sometimes major ones, that must be overcome in order to reach the goal. Occasionally the problems are so great that a project implodes but perhaps none so grandly as the $170 million debacle experienced by the Federal Bureau of Investigation in its effort to move from a paper and file cabinet system into the modern electronic world. As is the case with most projects gone awry, risk management lessons can be learned from a post mortem.

The Washington Post, which recently published a lengthy piece about the project, reported that the FBI had planned to build a new computer network, tie thousands of new personal computer work stations to the network and transition to a new software system that would be the nerve center of the system. The new network was to replace an old one that was based on mainframe technology (with the old green screen terminals) which was not even capable of storing photos or scanned reports. The old system was not readily available to everyone, with many employees having to rely on shared computers for internet access and e-mail. A task as simple as uploading one document took 12 steps, and many agents had stopped using the system and relied on paper and secretaries.

The federal commission that studied the September 11 attacks concluded that prior to September 11, “The FBI did not have an adequate ability to know what it knew.” The FBI opted to build a new, custom application to meet its needs.

The Post seems to conclude that the FBI did not effectively manage the project and that the vendor, with an open-ended contract and few safeguards, seemed willing to do the FBI’s bidding and bill out their time rather than tell the FBI that it was botching the project with its lack of management and indecision. Nor did the vendor, Science Applications International Corp. (SAIC) escape criticism for other shortcomings. The project ultimately was abandoned and the FBI has decided to start from scratch on a new project expected to cost $425 million.

The advice may seem like SAS (simple and stale), but when a project of that magnitude, with national security implications, goes that badly perhaps the advice cannot be repeated often enough. Judging by the Post’s report, it appears the following standards were not followed:

Projects require adequate and competent staffing on the part of both vendor and customer. If either party fails to devote enough people, who are competent and possess the kinds of skills required by the project, then the project will be handicapped. Significant personnel turnover, particularly in key positions, on either side can damage the effort. Utilizing different customer teams at the specification and testing stages can be hazardous. The testing team may have different agendas and motives than their own company’s design team.

Tremendous focus should be given, particularly by the customer, at the specification stage. Changes to the specifications after design/coding has begun often lead to the terrible trio: 1) costs go up, 2) work falls behind schedule and 3) the chances of bugs increase, which either creates functional problems or further exacerbates cost and schedule problems due to corrective efforts.

A strong change control process should be in place. Lax management by either side of the process may lead to higher costs which at the end of the day may result in an unhappy client. Changes should be justified in order to avoid unnecessary changes to the original specifications and the terrible trio.

Rigorous testing should be done before software is rolled into production and testing responsibilities should be clearly delineated. Failure to allocate responsibilities at the outset can result in finger-pointing later and that means conflict. Too often the only thorough testing comes when the software is brought into live production. By that time, it may be too late to avoid a disaster. What would have been mere effort to correct glitches now has an impact on the customer. It may be unable to perform for its own customers and its reputation with its customers or the general public may suffer as a result of poor performance due to system problems. Its revenues may decrease. It may even abandon the project and revert to the old one or look for another platform in desperation.

There are many other things an IT company can do to manage risk via carefully written contracts and a good insurance program. But the vendor also needs to remain diligent in applying common sense even to the point of telling its customer when the customer is not meeting the customer’s own obligations in the project – and documenting it.