Joseph Howerton

IS 430, Section 210 – David Bixby

Summer I - 2009-10

Case Study 1: FBI Virtual Case File (VCF) project

Introduction to the FBI’s VCF project failure

The FBI’s Trilogy IT modernization program was intended to upgrade/modernize the IT

Enterprise Architecture/Infrastructure of the FBI; by providing a high-speed network linking the

offices of the FBI, modern workstations and software within each office for every FBI employee,

and a user application known as the Virtual Case File to enhance the ability of agents to

organize, access, and analyze information. From A Review of the FBI’s Trilogy Information

Technology Modernization Program

Initial VCF project scope

It was the User Applications Component, which would ultimately become the VCF that staked

out the most ambitious goals. First, it was to make the five most heavily used investigative

applications—the Automated Case Support system, IntelPlus, the Criminal Law Enforcement

Application, the Integrated Intelligence Information Application, and the Telephone

Application—accessible via a point-and-click Web interface. Next, it would rebuild the FBI's

intranet. Finally, it was supposed to identify a way to replace the FBI's 40-odd investigative

software applications, including ACS. From A Review of the FBI’s Trilogy Information

Technology Modernization Program

Executive Summary

The back-story for the notorious FBI Virtual Case File (VCF) project failure really began in the

early 1990’s, when FBI Special Agent, Larry Depew, identified the need for a database program

that would help him (and other agents) to organize the reams of evidence collected and compiled

during their investigation(s); and from wiretaps, interviews, and financial transactions, over the

course of two and a half years investigating mafia involvement in skimming off millions of

dollars in federal and New Jersey state gasoline and diesel taxes. The problem that emerged was

one of an inability of FBI Special Agents to use a software component to connect the dots, both

on Agent Depew’s case specifically, and any new dots that might come along.

There was no doubt that a new system was needed, which would become the Virtual Case File

(VCF) system project; to replace the existing system, the archaic Automated Case Support

(ACS) system, as it was extremely cumbersome, inefficient, and limited in its capabilities. As it

turned out, Depew ended up writing a database program on his own, that he used to trace

relationships between telephone calls, meetings, surveillance, and interviews; but he still could

not import information from other investigations that would assist him in his own

investigation(s). That inability created the need for a new system which would result in the

formal creation of the FBI’s VCF project.

Officially, the FBI’s VCF project began in 2000 when the FBI began to deal with its outdated IT

systems infrastructure. On July 17, 2000, the FBI hired Bob E. Dies to create a plan for this IT

transformation. Dies, who was a former executive with IBM, basically replaced Louis J. Freeh,

who did not have the IT skills or competencies required to complete such a task. We will find

that the FBI’s lack of IT skills, a technical ignorance of technology, combined with their

incompetence in delivering IT systems infrastructure, is a common thread that binds this epic and

unnecessary IT project failure. And Dies was just the first of five ―officials‖ who, struggled over

four years to lead the FBI’s sprawling and antiquated information systems, and get the VCF

project under way.

In the fall of 2002, the only early warning sign came from Matthew Patton, a security expert

working for SAIC on the VCF development team. Patton unequivocally bashed the non-existent

technical expertise on the side of the FBI, as well as SAIC management practices, declaring both

organization’s incompetent. In the end of that scenario, Patton would not be granted security

clearance with the FBI, could not take a position with SAIC due to their inability to base him in

Chicago, and maybe most importantly, Patton’s declarations and objections were summarily

dismissed, thus ignored.

In 2005, Glenn A. Fine, the U.S Department of Justice’s inspector general, submitted an 81-page

audit of the FBI’s VCF project. In it he described eight factors that contributed to the VCF’s

failure. Among the factors included were poorly defined and slowly evolving design

requirements; overly ambitious schedules; and the lack of a strategic plan to guide hardware

purchases, network deployments, and software development for the bureau. Fine’s conclusion

was submitted in the light of the September 11th

, 2001, terrorist attacks on the World Trade

Center; he surmised that four years after 9/11, the FBI still did not have the software it needed to

―connect the dots‖ with the data in their case files.

The project was killed (before completion) in May 2005. The FBI spent $170 million on the

VCF project, and in the end they are going with a customizable, off the shelf system. The

solution is not to keep throwing money at the project, but to recognize the serious issues

surrounding the project’s failure. It is this researcher’s opinion that, with so much turnover at the

highest ranks of the FBI, and the inability to document lessons learned as part of an IT

Governance strategy, the FBI didn’t, and couldn’t have learned much.

In the end, this IT project, like so many IT projects, suffered from a total misunderstanding by

the FBI on how to define and implement Enterprise-wide architecture, based upon user

expectations. Every step of this project would have benefitted from defining and designing

around Usability Engineering tasks, activities, deliverables, and communication vehicles. It is

imperative that these practices occur from the beginning to the end of the project design and

development lifecycle.

Research Approach

Using IEEE Spectrum’s article on the VCF project failure as an anchor, I created a sort of outline

of the pertinent details that works for me. Then I went online to find further analysis on the topic.

I found that (because) the IEEE Spectrum article was (it seemed to me) the first to ―break the

story‖ in great detail to the general public, I felt it was used as the base anchor reference for

(basically) all of the analysis briefs out in the ether; so I figured that my analysis will be added to

the many.

As I read through the IEEE analysis, Goldstein pointed to another report of interest - A Review of

the FBI’s Trilogy Information Technology Modernization Program report on the FBI trilogy

Project, which changed my viewpoint. I found that this report is actually the most comprehensive

and complete report that I found on the failure of the VCF project. It details the issues across the

board, and puts forth detailed recommendations for the project based on their exhaustive

analysis.

From Anatomy of an IT disaster: How the FBI blew it

From here, I analyzed each of the other reference documents in deference to the expertise and

experience of the authors bringing it to the fore, having the core ideas resonate in a new way, to

capture their thoughts on an idea, when golden, and (of course) to find any differences or

discrepancies from opinion to opinion. The general consensus (I feel) in the research is one of

acknowledging precepts put forth in the IEEE Spectrum report that this enterprise project fell

into the most basic traps of software development, from poor planning to bad communications

throughout the lifecycle of the project.

Key Issues

The first key IT issue was the 800-page requirements document put forth by the FBI. This

document was totally unusable from the start, and many analysts assert that with such a poorly

defined approach to a project, left most to wonder, how could it possibly achieve success?

Further, the FBI’s understanding of the requirements was not strong at all. There was a seeming

disconnect between the FBI and SAIC from the beginning. Add to the mirth, a constant turnover

of FBI Directors, the FBI’s lack of internal expertise, and a lack of really any credible IT

management and technical expertise internal to the FBI. Of course the results would be (and

were) catastrophic for the VCF project.

The project demonstrated a systematic failure of software engineering practices which include,

from Wikipedia: Virtual Case File

Lack of a strong blueprint from the outset led to poor architectural decisions.

Repeated changes in specification.

Repeated turnover of management, which contributed to the specification problem.

Micromanagement of software developers.

The inclusion of many FBI Personnel who had little or no formal training in computer

science as managers and even engineers on the project.

Scope creep as the requirements were continually added to the system even as it was

falling behind schedule.

Code bloat due to changing specifications and scope creep. At one point it was estimated

the software had over 700,000 lines of code.

Violating Brooks' law by adding people and resources to the project as it was falling

behind, slowing it further.

Planned use of a flash cutover deployment, which made it difficult to adopt the system

until it was perfected.

Because of the bureaucracy (at least) inside the FBI (I think), as well as inconsistent responses

and communications from SAIC, there was no cogent, cohesive blueprint laid out for achieving

success. There was no enterprise architecture (blueprint) to describe how an organization

operates currently, how it wants to operate in the future, and includes a road map – a transition

plan – for getting there….leaving no detailed description of the FBI’s processes and IT

infrastructure as a guideline.

The National Research Council pointed out that without this blueprint/road map the bureau could

not make coherent or consistent operational or technical decisions about linking databases,

creating policies and methods for sharing data, and making trade-offs between information

access and security.

Key IT Impacts

According to the National Research Council’s (NRC’s) Report Review Committee in their report

A Review of the FBI’s Trilogy Information Technology Modernization Program, the specific

factors that would end up in an implosion of the VCF project include,

• Enterprise architecture - the committee concluded that the FBI’s efforts and results in the area

of enterprise architecture are late and limited, and fall far short of what is required.

• System design - The design process was well under way prior to the expansion of the

intelligence mission, and the requirements for the processes supporting the intelligence mission

were not included in the VCF design. For this reason, and because of the significant differences

in IT requirements between systems supporting investigation and those supporting intelligence,

the committee strongly recommends that the FBI refrain from using the VCF as the foundation

on which to build its analytical and data management capabilities for the intelligence processes

supporting the counterterrorism mission.

• Program and contract management - In practice, it is essentially impossible for even the most

operationally experienced IT applications developers to be able to anticipate in detail and in

advance all of the requirements and specifications. Therefore, internal development plans, and

the development contracts with supporting organizations, should call for an approach that is

based on a process of extensive prototyping and usability testing with real users. Doing so allows

iterative development with strong user feedback and involvement, thus increasing the chances

that what is ultimately delivered to the end users meets their needs.

• Skills, resources, and external factors - the FBI lacks a human resource and skill base adequate

to deal with the bureau’s IT modernization program. Specifically, the FBI is extremely short on

experienced program managers and contract managers and senior IT management team members

with good communications skills.

Final Analysis

It’s been a challenge just to figure out how to approach the analysis of the FBI’s failed VCF

project. It’s difficult to truly understand, with so many resources, and supposed expertise that

one would expect at their disposal, how this project could fail. I had to simplify how I was

looking at it. Upon reflection, my opinion is simply that the primary stakeholders on both sides

of the project turned out to be a bunch of people that didn’t have the guts to speak up and speak

out about the gross incompetence at any and every juncture of this project.

So, I approached my analysis of this colossal IT project failure wearing two hats; usability

engineer and project manager. My opinion paper begins and ends with Usability Engineering

(UE), as it is my opinion that the VCF project was a classic case for the need of insertion of UE

and User Experience Design (UXD) addressed across the project lifecycle to inform and speak for

the users. I just don’t see how they could execute this project so poorly. I know from my own

experiences that 2000-2005 were very tumultuous times for IT Governance boards, specifically

with the implosion (and consequences) of the internet bubble. I can imagine that SAIC was going

through a lot of not-so positive or pleasant experiences during those times as well.

In the final analysis, I assert that there is equal ―blame‖ here for both parties – the FBI, and SAIC

– to accept responsibility for the VCF project failure. I should say that I don’t really like the term

blame, except to say that when something goes wrong, per David Bixby, the IT Project Manager

should step out in front to shield the project team from said blame; I’m pretty confident that this

didn’t happen on either the side of the FBI or SAIC. Any IT project is going to have changes,

and requires constant communications, which I don’t feel was the case in this project. There

were issues from the very beginning that SAIC could have stopped the project in its tracks to

perform better, more comprehensive, up-front strategy.

The VCF has become a primary cautionary tale for IT professionals of all types. From the

perspective of a Usability Engineer, the lack of an appropriate created requirements document

left this project for dead from the beginning. In the end, I feel we are left wondering about too

many aspects of this IT project failure. As a result, along the way, along with major scope creep,

the FBI went through five different CIO’s, 10 Project Managers, and 36 contract changes. As

well as reports from the Government Accountability Office, the DOJ’s inspector general, and the

National Research Council, all submitted reports acknowledging the ongoing problems with the

project, as a result of not having an appropriate blueprint/roadmap to success.

I don’t feel that representatives from either side, the FBI or SAIC, stepped up and took any

responsibility for the ―comedy of errors‖ that the VCF became; and with the practice of Usability

Engineering taking root in IT development methodologies, a little up-front inspection, and with

checkpoint across the development lifecycle, this project could have been saved.

Instead, what the FBI and SAIC engaged the taxpayer in, was a project that was a roller coaster

of inefficiencies, riddled with change requests, constantly changing priorities in the guise of an

actual plan, schedule slippages, integration delays, the FBI’s sloppy inventories of existing

networks, underestimates throughout the development lifecycle, and significant management

turbulence. Throwing good money after bad to committee upon committee, and never realizing

the source of the problem, which was they employed no constructive usability methodologies to

appropriately engage the design and development of such an advanced Enterprise Architecture.

In the end, after spending at least $170 million for the VCF piece of the FBI’s Trilogy project,

the FBI is going to go with an off the shelf solution to address the requirements left unresolved in

the wake of the VCF project failure.

From a project management perspective, the lessons of the VCF cautionary tool are many. In the

light of day, the role of project manager from both sides of the project, was not integrated into

the project with any kind of authority.

Writing this brief has been very frustrating because the details of this project are enough to

frustrate any taxpayer alone, but also as an IT professional, with a focus in User Experience

Design and Project Management. In my opinion, it speaks to the ongoing corruption of

government; maybe especially if it’s a corruption of confidence; because that confidence has

slipped dramatically.

At every step of the way, when the FBI asked, Congress approved more and more money for the

Trilogy project. Where did all the money go for the VCF? The whole thing offends me quite

frankly. And I’m not asserting that I could have managed this, such a complex project. I am

simply saying that I received my BS in HCI in November 1999; so if this project started in 2000,

why weren’t there usability professionals assigned as resources to this project? It’s not clear to us

if there were or not, but I assert here that they could have saved this project at the beginning, by

rejecting the requirements documentation as put forth by the FBI.

The reason I’m being so vehement here is simply because if things can go this wrong when the

FBI is working on an IT project, what other projects are out there that are top secret, and we

know nothing of the waste and corruption, save what we hear from other government agencies

via committees and task forces after internal audits. It’s a bit nerve wracking when so much is

just unknown.

References

Who Killed the Virtual Case File?

http://spectrum.ieee.org/computing/software/who-killed-the-virtual-case-file

A Review of the FBI’s Trilogy Information Technology Modernization Program

http://www.enterprise-architecture.info/Images/Documents/CSTB-

FBI%20Project%20Review%2006-2004.pdf

Congressional Testimony of US DOJ inspector general Glenn A. Fine, February 2005

http://www.usdoj.gov/oig/testimony/0502/final.pdf

Matthew Patton's October 24, 2002 posting on InfoSec News about VCF

http://www.infosecnews.org/hypermail/0210/6688.html

The Failure Of The FBI’s Virtual Case File Project

http://strategicppm.wordpress.com/2010/04/05/the-fbis-virtual-case-file-project-and-project-

failure/

The FBI's Upgrade That Wasn't

http://www.washingtonpost.com/wp-dyn/content/article/2006/08/17/AR2006081701485.html

Anatomy of an IT disaster: How the FBI blew it

http://www.infoworld.com/d/developer-world/anatomy-it-disaster-how-fbi-blew-it-243

FBI's Virtual Case File Flops

http://www.internetnews.com/ent-news/article.php/3459921/FBIs-Virtual-Case-File-Flops.htm

Wikipedia: Virtual Case File

http://en.wikipedia.org/wiki/Virtual_Case_File

GAO: Questionable fees paid on FBI Trilogy project

http://gcn.com/articles/2006/05/08/gao-questionable-fees-paid-on-fbi-trilogy-project.aspx

Report: FBI wasted millions on 'Virtual Case File'

http://www.cnn.com/2005/US/02/03/fbi.computers/