CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
Transcript of CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 1/75
1 CREW is the acronym for Citizens for Responsibility and Ethics in Washington.
UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF COLUMBIA
____________________________________CITIZENS FOR RESPONSIBILITY AND )
ETHICS IN WASHINGTON, ))
Plaintiff, ))
v. ) Civil No. 07-01707 (HHK/JMF))
EXECUTIVE OFFICE OF THE )PRESIDENT, et al., )
)Defendants. )
____________________________________)
NATIONAL SECURITY ARCHIVE, ))
Plaintiff, ))
v. ) Civil No. 07-01577 (HHK)(JMF))
EXECUTIVE OFFICE OF THE )PRESIDENT, et al., )
)Defendants. )
____________________________________)
PLAINTIFF CREW’S1 OPPOSITION TO DEFENDANTS’
MOTION TO DISMISS
STATEMENT
In the fall of 2005, the White House discovered a large volume of emails were missing
from White House servers where they had been stashed in lieu of being archived in an electronic
record keeping system. On its own initiative the Office of Administration (“OA”) conducted an
in-depth analysis of the missing email problem, which involved a team of in-house and outside
contractors and was subject to a rigorous several-month independent validation and verification
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 2/75
2
process by an outside contractor. At the end of this process OA had documented evidence,
including a 250-page analysis, that the volume of missing emails may exceed 14 million.
With this extraordinary discovery in hand OA approached then-White House Counsel
Harriet Miers with a plan to restore the missing emails from back-up tapes created for disaster
recovery purposes. The response: deafening silence. The Bush White House initiated neither
OA’s recovery plan nor any other recovery plan, leaving the matter to languish for years. At the
same time, the White House continued its practice of storing emails in .PST files dumped on
White House file servers, even with proof positive of the risks of loss and destruction this
practice posed.
As for an electronic record keeping system, the Bush White House spent appropriated
funds to develop two different systems using outside contractors. When each of those systems
was ready for implementation the Bush White House once again decided to do absolutely
nothing, leaving in place an acknowledged deficient system for archiving all White House
emails. Apparently interest in at least one of the new systems waned after the White House
counsel was advised the system was “google-like” in its search capabilities.
In the spring of 2007, CREW disclosed the missing email problem, based on information
provided by confidential sources, in its report: Without a Trace: The Story Behind the Missing
White House E-Mails and the Violations of the Presidential Records Act . The White House
responded with widely conflicting statements, first admitting millions of emails were lost and
later denying any emails were missing at all. The subsequent filing of these lawsuits by CREW
and the National Security Archive in the fall of 2007 elicited no different response. The White
House continued to deny the existence of any email problem, argued against a requirement that it
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 3/75
3
preserve all backup tape copies of the missing email, and sought to prevent the Court from
reaching the merits of plaintiffs’ claims by asserting threshold legal defenses the Court
ultimately concluded lack merit.
Not until February 2008, in the face of an investigation and hearing by the House
Committee on Government and Oversight Reform, did the Bush White House finally
acknowledge it had begun to re-examine the missing email problem as part of a three-stage
process. Even then, however, White House officials testifying before the committee provided
few details and made clear the focus of their re-analysis was on finding flaws with the original
analysis rather than on ascertaining the true number of missing emails, the cause of the problem,
and the appropriate steps needed to rectify the problem and prevent its reoccurrence. The three-
step process was timed to be completed after the Bush administration left office, thereby limiting
public access to any restored presidential records.
One day after the Bush administration ended, defendants filed a motion to dismiss that
reflects an incredibly cynical and narrow view of defendants’ obligations under the Federal
Records Act (“FRA”). According to defendants, because they have taken some action -- no
matter how flawed, incomplete or limited -- the first four counts of plaintiffs’ complaints are
now moot. Hiding behind technical jargon and theoretical constructs, defendants attempt to
obscure three basic facts: we still do not know how many emails are missing; we still do not
know the source of the problem that caused emails to be missing in the first place; and we still
do not know if the problem has been fixed. That is because rather than measure what is missing
and compare that to what they have to answer the relevant questions of which emails are missing
and why, defendants adopted an approach seemingly designed exclusively to undermine the
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 4/75
4
results of OA’s earlier analysis. Toward that end defendants made certain assumptions not
grounded in fact, employed an abstract, highly restrictive theoretical methodology to identify
missing emails while ignoring the full inventory of actual emails contained on the backup tapes,
and inexplicably decided to restore missing emails from less than 50 percent of the days even the
most recent analysis identified as missing significant numbers of emails. Not surprisingly,
defendants have not and cannot say they now have a complete set of emails from the Bush
presidency.
Of equal concern, defendants ask the Court to dismiss four claims based on a severely
restricted record that omits the most fundamental facts. For example, defendants rely on the
work of an unidentified contractor performed at an unidentified time and about which an OA
official who was with OA for only four months and had no apparent expertise has proffered only
scant details. Defendants claim repeatedly their expenditure of $10 million justifies dismissal,
but offer no explanation or documentation of how that money was spent. In describing the third
and final phase of their process defendants allude suddenly to a “prior analysis” from which they
identified an addition 125 missing .PST files, but provide no details about this analysis or even
what volume of missing emails are contained in those 125 .PST files. These are just some of the
critical facts completely missing from the record and without which the Court cannot possibly
conclude defendants have met their legal obligations so as to moot the first four claims.
At bottom, defendants’ latest motion to dismiss is yet another gambit in a series of
actions designed to avoid transparency and accountability by obscuring the fact the Bush White
House did nothing for years about a serious email problem that left a gaping hole in our nation’s
history. Indeed, without any information about what caused the emails to go missing in the first
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 5/75
2 The FRA defines “record” to include:
all books, papers, maps, photographs, machine readablematerials, regardless of physical form or characteristics,made or received by an agency of the United StatesGovernment under Federal law or in connection withthe transaction of public business and preserved or
5
place, we have no assurance the problem will not be repeated. With the recovery process far
from complete, defendants want to go no further. But the claims in this case are not moot,
requiring the Court to deny defendants’ motion to dismiss.
STATUTORY BACKGROUND
The Federal Records Act
The FRA is a collection of statutes governing the creation, management and disposal of
federal records. See generally 44 U.S.C. §§ 2101 et seq., 2901 et seq., 3010 et seq., and 3301 et
seq. Among other things, the FRA ensures “[a]ccurate and complete documentation of the
policies and transactions of the Federal Government,” as well as “judicious preservation and
disposal of records.” 44 U.S.C. § 2902; Armstrong v. Executive Office of the President, 1 F.3d
1274, 1278 (D.C. Cir. 1983) (“Armstrong II).
To fulfill this purpose, the FRA requires the head of each agency to “make and preserve
records containing adequate and proper documentation of the organization, functions, policies,
decisions, procedures, and essential transactions of the agency.” Id. at § 3101. Under the
statute, each agency must also “establish and maintain an active, continuing program for the
economical and efficient management of the records of the agency,” id. at § 3102, and must
“establish safeguards against the removal or loss of records” the agency head determines are
necessary and required by regulations of the archivist. Id. at § 3105.2
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 6/75
appropriate for preservation by that agency . . as evidenceof the organization, functions, policies, decisions, proceduresoperations, or other activities of the Government or becauseof the informational value of data in them.
44 U.S.C. § 3301.
6
Documents that satisfy the Act’s definition of “record” can be disposed of only under the
provisions of the FRA. 44 U.S.C. § 3314. Pursuant to the Act’s records disposal process, an
agency may dispose of a record only upon authorization from the archivist based on the agency’s
determination it no longer needs the record and the archivist’s determination the record does not
have “sufficient administrative, legal, research, or other value to warrant [its] continued
preservation by the Government.” 44 U.S.C. §§ 3303a(a), (d) and (e). In addition agencies are
permitted to destroy “certain common types of records pursuant to disposal schedules
promulgated in advance by the Archivist.” Armstrong II, 1 F.3d at 1279.
An agency wishing to dispose of records must first submit to the archivist either lists of
records the agency head determines “are not needed by it in the transaction of its current
business,” 44 U.S.C. § 3303a(2), or “schedules proposing the disposal after the lapse of specified
periods of time of records of a specified form or character” the agency head determines will not
have “sufficient administrative, legal, research, or other value to warrant their further
preservation by the Government.” Id. at § 3303a(3). Upon receipt of such a request, the
archivist must issue a notice requesting public comment on the agency’s proposal and the
archivist’s staff must conduct its own assessment of the value of the records the agency is
proposing to destroy. Id. at § 3303a(a). The archivist is free to accept or reject an agency’s
proposal. Id. In addition, the archivist may request advice and counsel from Congress regarding
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 7/75
7
the disposal of any federal records the archivist believes may be of special interest to Congress
or where the disposal of those particular federal records is in the public interest. 44 U.S.C. §
3301a(c).
The FRA also establishes a remedial scheme governing the improper removal or
destruction of a record from an agency. This scheme reflects Congress’ recognition of “the need
for devising adequate statutory safeguards against the unauthorized removal of agency records . .
.” Kissinger v. Reporters Comm. for Freedom of the Press, 445 U.S. 136, 149 (1980).
Accordingly, if the archivist learns of “any actual, impending, or threatened unlawful removal . .
. or destruction of records in the custody of [an] agency,” the archivist is required to notify the
head of the agency and assist the agency head “in initiating action through the Attorney General
for the recovery of records wrongfully removed and for other redress provided by law.” 44
U.S.C. § 2905(a). If the agency head does not initiate such action, the archivist “shall request the
Attorney General to initiate such action, and shall notify the Congress when such a request has
been made.” Id.
The FRA places a similar and independent duty on the head of each federal agency to
“initiate action through the Attorney General for the recovery of records he knows or has reason
to believe have been transferred to his legal custody.” 44 U.S.C. § 3106. If the agency head
refuses to pursue legal remedies, the Archivist -- as discussed above -- must request that the
attorney general take action and must inform Congress he has made this request. Id.
The archivist also has an affirmative duty to guide and assist federal agencies to ensure
the adequate and proper documentation of the policies and transactions of each agency. 44
U.S.C. § 2904(a). In furtherance of these duties, the archivist must promulgate standards and
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 8/75
3 The complaints of CREW and the National Security Archive are nearly identical anddefendants have filed a single motion to dismiss both complaints. This opposition addressesonly CREW’s complaint and accordingly references herein to “plaintiff” refer only to plaintiff CREW and not the National Security Archive, which will be responding separately todefendants’ motion.
8
guidelines for federal agency records management, must establish standards for selective
retention of records, and must assist the agencies in applying the standards to records in the
agencies’ custody. 44 U.S.C. §§ 2904(b), (c)(1), 2905(a).
At the same time, heads of federal agencies are under an affirmative duty to implement
the standards and guidelines for the retention of federal records. Specifically, each agency head
must maintain an active records management program that provides for effective controls over
the creation and use of federal records and that ensures the application of the archivist’s
standards and procedures for the preservation of federal records. 44 U.S.C. § 3102. In addition,
agency heads are required to establish safeguards against the removal or loss of records
determined by the agency to be necessary and required by regulations of the archivist. Id. at §
3105.
This Litigation
CREW’s eight-count complaint3 stems from the failure of the Executive Office of the
President (“EOP”), OA and its head, the archivist and the National Archives and Records
Administration (“NARA)” to take any action in the face of knowledge that millions of federal
and presidential records have been deleted. Memorandum Opinion and Order, Nov. 10, 2008
(Document 90) at p. 6. CREW also challenges defendants’ failure to adopt an adequate system
to preserve such electronic records. Id. Claims One through Four of CREW’s complaint,
requesting declaratory and mandamus relief, seek to compel defendants to activate the remedial
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 9/75
4 Nat’l Cmty. Reinvesment Coal. v. Nat’l Credit Union Admin., 290 F.Supp.2d 124, 131(D.D.C. 2003).
9
enforcement scheme of the FRA through relief requiring the archivist to take action to recover
the missing emails and the White House to request the attorney general to take recovery action.
Id.
Claims Four through Eight seek to compel defendants to establish an appropriate records
management system for archiving and preserving White House emails. Id. Toward that end
defendants request injunctive and mandamus relief compelling defendants to fulfill their duties
under the FRA by establishing such a system.
ARGUMENT
STANDARD ON MOTIONS TO DISMISS
While ruling on a motion to dismiss for lack of jurisdiction brought under Fed. R. Civ. P.
12(b)(1) the Court is free to consider “relevant materials outside the pleadings,”4 but still must
construe the complaint “‘liberally, and the plaintiff should receive the benefit of all favorable
inferences that can be drawn from the alleged facts.’” Sierra Club v. Mainella, 2005 U.S. Dist .
LEXIS 18911, *13 (D.D.C. 2005), citing EEOC v. St. Francis Xavier Parochial Sch., 117 F.3d
621, 624 (D.C. Cir. 1997). Accordingly, defendants here should not prevail on their motion to
dismiss “unless Plaintiffs can prove no set of facts in support of their claim which would entitle
them to relief.” Kowal v. MCI Commc’n Corp., 16 F.3d 1271, 1276 (D.C. Cir. 1994).
Moreover where, as here, the “jurisdictional facts” defendants have identified are
“inextricably intertwined with the merits of the case [the court] should usually defer its
jurisdictional decision until the merits are heard.” Herbert v. Nat’l Acad. of Sci., 974 F.2d 192,
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 10/75
10
198 (D.C. Cir. 1992), citing Land v. Dollar, 330 U.S. 731 (1947). In such cases, where the court
is asked to look “beyond the pleadings” it “must bear in mind what procedural protections could
be required to assure that a full airing of the facts pertinent to a decision on the jurisdictional
question may be given to all parties.” Id. Accordingly, “ruling on a Rule 12(b)(1) motion may
be improper before the plaintiff has had a chance to discover the facts necessary to establish
jurisdiction.” Id.
As set forth below, for these very reasons this Court must either dismiss defendants’ Rule
12(b)(1) motion outright or, at a minimum, defer ruling on the motion in light of the contested
facts and plaintiff’s inability to date to discover facts necessary to resolve the question of
whether plaintiff’s claims are moot.
I. DEFENDANTS HAVE NOT DEMONSTRATED BY UNDISPUTED
FACTS PLAINTIFF’S FIRST FOUR CLAIMS ARE MOOT.
A. The Limited Actions Defendants Have Taken To Date Do Not Moot
Plaintiff’s Claims.
Defendants’ motion to dismiss rests on their claim that since the filing of this lawsuit,
they have taken all the action the FRA requires of them thereby rendering the first four claims of
the complaint moot. As a line of Supreme Court decisions establishes, however, “the ‘voluntary
cessation of allegedly illegal conduct does not deprive the tribunal of power to hear and
determine the case, i.e., does not make the case moot.’” DeFunis v. Odegaard, 416 U.S. 312,
318 (1974), citing United States v. W.T. Grant Co., 345 U.S. 629, 623 (1953); United States v.
Trans-Missouri Freight Ass’n, 166 U.S. 290, 308-310 (1897); Walling v. Helmerich & Payne,
Inc., 323 U.S. 37, 43 (1944); Gray v. Sanders, 372, U.S. 368, 376 (1963); United States v.
Phosphate Export Ass’n, 393 U.S. 199, 202-03 (1968). Instead, where jurisdiction is “properly
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 11/75
5 Mr. Everett therefore cannot properly be deemed an expert witness.
11
acquired” at the outset the case will become moot only when two conditions are met: (1) “it can
be said with assurance that ‘there is no reasonable expectation . .’ that the alleged violation will
recur” and (2) “interim relief or events have completely and irrevocably eradicated the effects of
the alleged violation.” County of Los Angeles v. Davis, 440 U.S. 625, 631 (1979) (citations
omitted). Neither condition is met here.
As an initial matter, there is no question jurisdiction was “properly acquired” at the outset
of this litigation, as this Court’s Memorandum Opinion and Order of November 10, 2008
confirms. Indeed, defendants do not dispute they had taken no action with respect to the missing
emails when the complaint was filed. Instead, they argue the three-phase analysis they
conducted over the past several months constitutes sufficient agency action and therefore
requires dismissal of plaintiff’s first four claims.
At bottom defendants’ argument is really far more limited, namely that having identified
flaws with the 2005 analysis that translate into a different number of missing emails than initially
discovered, they have met their legal obligations under the FRA. In support defendants offer the
declaration of OA Chief Information Officer Stephen M. Everett who, after a mere four months
in his position and with no claimed expertise of any kind,5 offers a general description of OA’s
most recent analysis as well as his limited understanding of the analysis OA conducted in 2005,
focusing most specifically on the flaws in the 2005 analysis.
The backdrop to both analyses is the admittedly imperfect system the Bush White House
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 12/75
6 For example, an internal EOP document dated October 25, 2005, and produced by theHouse Committee on Oversight and Government Reform (attached as Exhibit 1) details the“operation risk in current email storage management processes.” Those risks include lost ormisplaced email.
7 Indeed, it may continue to the present day as the record before the Court reflects nochanges to the Bush White House’s practice of archiving emails in .PST files stashed on WhiteHouse servers.
8 See Letter from Patrick J. Fitzgerald to Counsel in United States v. I. Lewis Libby,January 23, 2006 (attached as Exhibit 2). Further documentation provided by the HouseCommittee on Oversight and Government Reform and attached as Exhibit 3 explains OA’sunsuccessful efforts to recover Office of Vice President (“OVP”) email for the period September30, 2003 to October 6, 2003. Both the files on the file servers used to store .PST filesand the
journal mailboxes restored from a backup of the server contained no messages for the targetperiod. In the end, some emails were located after restoring the Exchange server containingOVP mailboxes from a backup and extracting email from individual OVP mailboxes.
12
used to archive emails.6 Intended to be a stopgap measure during the transition from Lotus
Notes to the Microsoft Exchange email system, the storage of archived emails in .PST files on
White House servers became the permanent archival method for the duration of the Bush
presidency.7 At least one known consequence was the inability of the White House to produce to
Special Counsel Patrick J. Fitzgerald all relevant and responsive email of the EOP and the Office
of the Vice President.8
Defendants have now confirmed through their 2008 analysis that a much larger volume
of emails also was missing. The loss of email verified by the most recent analysis also confirms
the ineffectiveness of the archiving procedure the White House has been using since the
migration from Lotus Notes to Microsoft Exchange. What remains unknown -- because of the
flaws and limited scope of that analysis -- are whether all White House emails during the Bush
presidency have now been accounted for, whether the cause of the missing emails has been
identified, and whether the problem has been fixed. In other words, both the scope of the
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 13/75
9 This partial restoration process also demonstrates missing emails can be recoveredsuccessfully from backup tapes.
10 Declaration of Stephen M. Everett (“Everett Decl.”) p. 1 (Document 112-2).
13
problem and its root cause have yet to be determined, information linked inextricably with
ascertaining what emails are missing. At bottom we are left simply with the identification of
flaws in the 2005 analysis and the partial restoration of a small subset of missing emails,9 but
these do not a moot case make.
Mr. Everett’s declaration adds very little to this picture. His understanding of the 2005
analysis is based on discussions with unidentified staff of his office and “documents presented to
[him],”10 none of which he identifies further. See, e.g., Everett Decl. at p. 3 (“According to
OCIO documents . . .”). Also unknown is whether any of the staff who provided information to
Mr. Everett participated in the 2005 analysis or were even employed at OA while the earlier
analysis was conducted. Mr. Everett’s declaration contains similarly vague allusions to other
knowledge OA has acquired about the 2005 analysis with no identification of the source. For
example, in attempting to discredit the 2005 analysis Mr. Everett states: “We also learned that
the tool used to ‘count’ messages in the 2005 effort was limited.” Id. at 4. Nowhere, however,
does Mr. Everett explain the origin of this information.
Mr. Everett also describes certain assumptions OA made about the 2005 analysis, but
provides no explanation of the basis for these assumptions. For example, he states “the 2005
effort evidently attempted to identify the number of e-mail messages preserved by the various
EOP components on specific dates by counting the number of email messages contained in .PST
files,” id. (emphasis added), but does not explain why this “evidently” is the case. Given that the
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 14/75
11Additional documentation includes a 250-page analysis that defendants to date haverefused to produce.
14
only documentation defendants include from the 2005 analysis is a color-coded chart,11 it is
impossible to verify the accuracy of this or any other of Mr. Everett’s assumptions.
Mr. Everett’s description of the so-called “Three-Phase Email Restore Process” is
similarly deficient, relying as it does on his review of unidentified documents described only as
“relevant documentation” and discussions with unidentified staff about a process that took place
before he came to OA. Id. at 5. As a result he is reduced to describing what he has “been made
aware of” and his “understanding” of various aspects of the process, Everett Decl. at 7, with no
first-hand knowledge whatsoever. See, e.g., id. at 9 (“It is my understanding that the ARIMA
model was used to identify days or groups of days with statistically significant low e-mail counts
. . .”). Indeed, at no point does Mr. Everett state he has ever spoken personally to or met with
either the unidentified third-party vendor OA contracted with in August 2008 to complete Phase
III or Dr. Nancy J. Kirkendall, identified as the individual who developed the ARIMA model for
OA.
Perhaps in recognition of the glaring deficiencies in OA’s latest analysis, Mr. Everett
ultimately falls back on the truism one cannot ever know with absolute certainty “how many and
what precise e-mails there should in fact be.” Everett Decl. at 12. From this he asks the Court to
accept his blanket, and as yet unproven assertion, “OA has completed all that reasonably may be
done to restore any records that may exist that are potentially not contained in the e-mail
message archives by using the disaster recovery back up tapes.” Id. For the Court to reach this
conclusion based on the record before it, however, would be plain error.
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 15/75
12 County of Los Angeles v. Davis, 440 U.S. at 631.
15
First, as described above, Mr. Everett has provided only the most cursory of explanations
of what OA has done based on unidentified sources and documents and a glaringly incomplete
factual record. With no opportunity to test the accuracy of any of his statements, neither the
Court nor the plaintiffs can confidently conclude OA has met its legal obligations.
Second, what little defendants have offered does not prove their central assertion they
have taken all reasonable steps to locate and restore missing emails. Most fundamentally, while
defendants have now confirmed there is, in fact, a missing email problem, they have yet to
identify three critical facts about that problem. First, defendants have not identified the total
volume of missing email, focusing instead on the alleged problems with the number of email
identified as missing by the 2005 analysis. Second, defendants have said absolutely nothing
about the cause of the missing email problem, a fundamental prerequisite to ensure both that the
problem will not recur and that “interim relief or events have completely and irrevocably
eradicated the effects of the alleged violation.”12 And third, defendants have not stated whether
the problem causing the missing email has been fixed, again leaving the Court no assurance the
problem will not recur and the steps defendants have taken fully address the effects of their prior
inaction. Without these essential facts the Court cannot conclude the first four claims are moot.
Third, defendants fail even to acknowledge, much less address, the underlying problems
with the EOP’s email archiving system. While Mr. Everett offers a two-paragraph description of
the system accompanied by a one-page graphic, he omits critical details. An internal OA
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 16/75
13 Of note, defendants did not examine any data after August 2005, even though therelevant period alleged in the complaint continues through October 2005. One consequence of this omission is that any missing emails related to OA’s discovery of the email problem inOctober 2005 will not be restored.
14 This document was included in the exhibits to a hearing held in February 2008 by theHouse Oversight and Government Reform Committee. In addition, former OA employee StevenMcDevitt submitted answers to written questions from the Committee (attached as Exhibit 4)that contain a description of four risks identified within OA from the lack of an “adequate emailrecords management solution in place.” Id. at 7. These risks include incomplete data, aninability to demonstrate the data in the system is complete, public perception problems in light of the email problems that occurred during the Clinton administration, and the lack of useraccountability, meaning “[n]o verification that data retained has not been modified or whatactivities have been performed by system users or administration.” Id.
16
analysis prepared in October 2005,13 when the missing email problem was first discovered,
documents multiple problems with the Bush White House’s email archive process, none of
which is mentioned by Mr. Everett. They include the following:
C Dependency on manual operations and monitoring, with institutionalknowledge held by a single administrator;
C The non-existence of standard operating procedures for emailmanagement;
C Lack of robust automated tools to support the email archive process,including that the then-current version of Mail Attender was prone tofailure;
C
Lack of monitoring of archive processes;
C Lack of a dedicated archive location, resulting in emails stored in variouslocations that are created only when there is available space.
Email Archive Process Risk Mitigation, Oct. 25, 2005 (Exhibit 1).14 These operational risks
were flagged as likely to lead to lost or misplaced email archives. Id. And that is exactly what
resulted from the Bush White House’s continued reliance on a fundamentally flawed email
archiving system.
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 17/75
15 Ironically, while defendants fault the 2005 analysis for its inability to allocateapproximately 10 million email messages to a specific component, they admit their purportedlymore sophisticated and accurate PIVIT tool still leaves “roughly 12 million messages”unaccounted for. Ds’ Mem. at 21.
16 Defendants describe their ARIMA model as used to analyze “large data pools withtime-series components,” citing to its use in other contexts to predict electricity prices andforecast sugar cane production. Ds’ Mem. at 22 and n.8. But those are situations where varyingseasonal weather conditions have a major impact on the outcome. In any event, as evendefendants concede, the numbers resulting from their application of the ARIMA model are notaccurate. See Ds’ Mem. at 23 n.9. Moreover, defendants justify using the ARIMA modelbecause of the purported failure of the 2005 analysis to account for “weather-related closures andholidays, when ‘low counts’ or ‘zero days’ might be expected . . .” Ds’ Mem. at 15. To thecontrary, as former OA official Steven McDevitt explained, the 2005 analysis used a statisticalprediction “based on a six-week moving average that separated working days from non-workingdays” (such as weekends and federal holidays), that “also accounted for the natural seasonalfluctuations in email volume” such as “when the President is on vacation,” and “allowed forspikes in email volume as a result of world events.” Responses from Steven McDevitt, question
24, p. 6 (Feb. 21, 2008).17 At least some of the inconsistencies between the 2005 and 2008 analyses suggest
problems may lie with the most recent analysis. Defendants’ comparison chart (Document 112-4) reveals days where the 2005 count -- claimed to have undercounted emails -- actually includesa greater number of emails. For example, the chart lists the 2005 day sum for October 27, 2003as 433,421, while the Phase III count for that same day is 151,619. Document 112-4 at p. 8.
17
This case flows from the central proposition that in the face of evidence suggesting
millions of emails were missing from White House servers defendants did nothing. While
defendants have now done a little more than nothing, they still have not measured and quantified
the precise problem. Instead, defendants have focused on the 2005 analysis rather than the
overall problem of missing emails and, based on assumptions, handy technological gadgets (e.g.
the “PIVIT,” which defendants claim was “independently verified as an accurate tool” by an
unidentified contractor using unidentified methods of verification),15 and an abstract statistical
model (the ARIMA) that has no relation to what actually goes on in the White House,16 have
attempted to identify a host of problems with that analysis.17
But any flaws in the 2005 analysis
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 18/75
18
Indeed the 2008 analysis lacks the granularity allegedly missing from the 2005analysis. For example, it looks at emails on a per component basis only. As a result, if emailsconsistently are missing from particular users (e.g. the vice president or Karl Rove) but thosegaps do not result in an overall “low” count for a particular component, the loss will never beidentified.
19 Responses from Steven McDevitt at p. 7, question 11.
18
still do not answer the questions of how many emails are missing and why.18 As a result,
defendants do not and cannot say they have found or otherwise accounted for all emails.
Defendants’ restoration process also suffers from fatal defects. Most fundamentally, it
restored emails only from selected Exchange Journal Mailboxes on backup tapes, adhering to the
system the Bush White House adopted for storing emails, see Everett Decl. at 2, and ignored the
full complement of emails residing on the full Exchange server backup tapes, which by
definition would have contained all the email contained in the Exchange email systems at the
time the backup was created. Defendants have identified no basis on which they could
reasonably conclude the journaling process functioned properly so as to justify their restoration
of emails found only on Exchange Journal Mailboxes on the backup tapes. Indeed, Mr.
McDevitt described the archiving process the White House was using under Microsoft Exchange
as “primitive” and posing a high risk data would be lost.19 Yet with no validation of the
journaling process, defendants nevertheless limited their restoration efforts to selected journaled
mailboxes and ignored the more complete cache of emails residing on the full Exchange server
backups.
In addition, the restoration is based on email counts identified by the ARIMA model.
Putting aside its clear flaws, the ARIMA model still identified at least 106 component days with
statistically low email counts. Yet inexplicably defendants chose to restore emails for only 48
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 19/75
20 In addition, the 2008 analysis examined dates ranging between January 1, 2003 andAugust 10, 2005, even though the alleged period in question extends through October 2005.
21 By contrast, the Court and the public had far more knowledge about the emailmalfunction that occurred during the Clinton administration. As outlined in the GAO report,Electronic Records: Clinton Administration’s Management of Executive Office of thePresident’s E-Mail System (available at http://www.gao.gov/new.items.d01446.pdf), missingemails were attributed to anomalies with incoming email to users on a specific White Houseserver, anomalies with incoming email to users with first names beginning with the letter D, anda period of time during which the Office of the Vice President failed to preserve backup tapes.With these problems identified, the Clinton administration was able to identify and restore allmissing emails and ensure the problem did not recur.
19
component days, leaving unaddressed more than 50% of the component days identified by the
ARIMA model as statistically low. See Ds’ Mem. at 25.20 Moreover, when they filed their
motion to dismiss defendants had not completed even this limited restoration. As a result, there
is no way to ascertain the actual accuracy of the theoretical counting process defendants have
employed. And with absolutely no information about what caused the problem in the first place,
the Court can have no assurance it has been fixed and will not recur.21
B. The Efforts Of Crew And Congress To Redress The Missing White
House Email Problem Do Not Satisfy Defendants’ Affirmative
Obligations Under The FRA To Initiate Enforcement Action.
Defendants also make the astonishing argument the first four claims were rendered moot
by CREW’s request of the attorney general to initiate an investigation as well as the
congressional investigation conducted by the House Committee on Oversight and Government
Reform. Ds’ Mem. at 26 n.11. The outcome of that request and hearing -- absolutely no change
in the status quo -- illustrates perfectly why the statutory scheme imposes on the defendants far
more robust responsibilities than defendants acknowledge. Simply stated, a request from either
the White House or the archivist is far more difficult to ignore than a request from an outside
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 20/75
22 Perhaps that is why the Bush White House took pains to conceal from NARA criticalinformation about its email records management. See, e.g., GAO Information Management:The National Archives and Records Administration’s Fiscal Year 2008 Expenditure Plan(Document 95-2) at 5 (noting the “ongoing uncertainty about the format and volume of recordsto be transferred” from the Bush White House to NARA at the end of the administration);
Responses from Steven McDevitt, question 30, p. 8 (describing how during the summer of 2006he was “directed by the CIO that I was not allowed to discuss the potential email retention issuesand the analysis that was performed by OCIO with the NARA staff” and that he was “to informany NARA staff who contacted me about these issues to direct all inquiries about email recordsmanagement to White House Counsel and White House Records Management.”).
23 Ds’ Mem. at 26 n. 11.
20
non-governmental group such as CREW.22
As this Court already has concluded, under the FRA both the agency head and archivist
“plainly have a duty to take enforcement action when necessary.” Memorandum Opinion and
Order, Nov. 10, 2008, at 21.
[T]he decision to seek the initiation of an enforcement actionto prevent the destruction or removal of records is not com-mitted to the agency head’s or Archivist’s discretion . . .Because the FRA enforcement provisions leave no discretionto determine which cases to pursue, the agency head’s andArchivist’s decisions are not committed to agency discretionby law. In contrast to a statute that merelyauthorizes anagency to take enforcement action as it deems necessary, the
FRA requires the agency head and Archivist to take enforce-ment action.
Id., quoting Armstrong v. Bush, 924 F.2d 282, 295 (D.C. Cir. 1991) (emphasis in original).
Passively sitting back because the attorney general “has been made aware” of missing records
claims through other means23 simply is not an adequate substitute for the affirmative obligations
the FRA imposes on both OA’s head and the archivist.
It is undisputed defendants here have never initiated enforcement action to prevent the
destruction of White House emails. Having failed to fulfill their mandatory obligations under the
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 21/75
21
FRA defendants cannot evade responsibility by relying on the efforts of external groups like
CREW and Congress. As a result, those efforts do not render plaintiff’s claims moot.
In sum, the mere fact defendants have taken some very narrow and limited action, after
sitting idly by for years, does not satisfy their affirmative obligations under the FRA to initiate
enforcement action to prevent the destruction of agency records. As even defendants’ most
recent analysis confirms, agency records are missing and, with no understanding of the cause or
how to fix it, the risk of further destruction remains. Ignoring the statutory scheme altogether,
defendants have cast themselves as the proverbial fox guarding the henhouse door and argue in
essence no outside involvement by NARA or the attorney general is necessary because the White
House -- the very entity that created the problem in the first place -- is comfortable with its
analysis of and remedy for the missing email problem. But that analysis is so manifestly
inadequate as to leave the most fundamental questions raised by plaintiff’s complaint
unanswered. And defendants’ purported “remedy” is so manifestly deficient as to provide no
assurance the vast majority of missing emails have been found and restored. Accordingly,
plaintiff’s claims cannot properly be considered moot.
II. DEFENDANTS’ ATTEMPT TO RECAST PLAINTIFF’S
CLAIMS AS CHALLENGING UNREVIEWABLE AGENCY
ACTION CANNOT SUCCEED.
Failing to establish the mootness of plaintiff’s inaction claims, defendants argue those
claims have now morphed into agency action claims for which the FRA does not authorize
review. This argument, however, reflects a fundamental misunderstanding of the nature of
plaintiff’s claims and the limits of the Court’s review.
As discussed above, defendants do not and cannot dispute their failure to trigger the
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 22/75
22
enforcement mechanisms of the FRA through requests of the attorney general to initiate action to
recover the missing and deleted emails. This failure lies at the core of plaintiff’s “inaction”
claims and the declaratory and mandamus relief plaintiff seeks is aimed precisely at triggering
this process.
Nothing about defendants’ latest “action” alters this status. That action confirmed emails
are missing and there remains a risk of further document destruction, but nevertheless did not
include a request of the attorney general to initiate action. Instead, defendants have operated
completely independent of the FRA’s administrative enforcement scheme. That is why, as
explained, above, plaintiff’s claims seeking to trigger that scheme are not moot.
In arguing to the contrary defendants essentially suggest that administrative process is
now unnecessary because they have taken some steps in an attempt to resolve the missing email
problem. At the same time, they argue the Court cannot evaluate anything about the
appropriateness or merits of the steps they have taken in order for the Court to determine
whether the administrative enforcement scheme is now, in fact, unnecessary. According to
defendants, such an evaluation “would amount to an impermissible attempt to ‘invoke the
federal courts to prevent an agency official from improperly destroying or removing records.’”
Ds’ Mem. at 30, citing Armstrong v. Bush, 924 F.2d at 294. In other words, defendants argue,
the Court (and plaintiff) must take at face value defendants’ claims that the action they have now
taken eliminates any need for the attorney general to separately assess whether further recovery
efforts are warranted.
Accepting defendants’ argument would completely eviscerate the role of the Court and
nullify the D.C. Circuit’s express recognition of the right of private plaintiffs like CREW to
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 23/75
23
“bring suit to require the agency head and Archivist to fulfill their statutory duty to notify
Congress and ask the Attorney General to initiate legal action.” Armstrong v. Bush, 924 F.2d at
295. Any such suit necessarily is premised on a claim that the agency head and/or archivist
failed to trigger the enforcement mechanism of the FRA. Evaluation of the merits of such a
claim -- a role Armstrong clearly provides for the Court -- will, in cases such as this one, require
the Court to evaluate whether any actions the agency has taken excuse its failure to notify
Congress or seek the initiation of a legal action through the attorney general. Indeed, the case
defendants present in support of their dismissal motion focuses explicitly on the three-phase
process they have undertaken. Without the ability to assess defendants’ actions for purposes of
determining whether defendants have failed to fulfill their statutory obligations, plaintiff’s
judicially recognized right to bring these claims in the first place and the Court’s judicially
recognized right to review such claims are rendered a nullity.
The cases defendants cite do not provide to the contrary, as their holdings flow from a
plaintiff’s attempts to broaden their claims of injury in resistence to a mootness claim. See, e.g.,
Clarke v. United States, 915 F.2d 699, 701 (D.C. Cir. 1990) (cited in Ds’ Mem. at 28). Here, by
contrast, plaintiff is not claiming any additional or broadened injury. Indeed, defendants’ most
recent analysis confirms the underlying need for the declaratory and mandamus relief plaintiff
seeks and the injury it has claimed, namely that emails are missing with no assurance the
problem is not continuing today.
CONCLUSION
For the foregoing reasons, defendants’ motion to dismiss should be denied. Pursuant to
LCvR 7(f), CREW respectfully requests an oral hearing on this matter.
Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 24/75
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 25/75
EXHIBIT 1
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 26/75
Email Archive ProcessRisk Mitigation
.
Discussion Docu¡'nent
October 25, 2005
lil,l
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 2 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 27/75
There is operational risk in current email storage managëment processes. Lost or misplaced email
archives may result in an inability to meet statutory reeyi¡ements.
This risk can be mitigated. However, several steps must be taken over time to close exposures torisk.
'
lmplement procedures.Update the Mail Attender tool.lmplement system monitoring.lmplement a dedicated email archive. ,
',
Upgrade Exchange serverr¡ to Exchange 2003. .
.t
Senior management authorization is required to proceed:with risk mitigation recommendations.
There are significant benefits to the recommendations. i ,
Reduces the frequency of enor typical of,manual processes.Gloses exposure to risk by isolating the email archive from data and file storage.Provides safeguards to assure the integrity of the þrocess.Manages data with vendor supported technologie$.'l'
.;i,
For Official Use Only
Executive Suñmary
voloIoc¡ululN
2
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 3 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 28/75
There is operational risk in current email storagemanagement processes.
¡-ov,fov,oct¡,
=o
ct)cCDtsa(E9.Oií r-
oðoøìL- t..eq.È, ft(U È¡outË..8cLt<5
For Official Use Only 3
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 4 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 29/75
Lost or misplaced email archives may result in an
..'The email archive process depends on manual òperations and monitoring.
Institutional knowledge is held be a singte administratorThe administrator must launch each procesq and monitor to completion
Standard operating procedures for email management do not exist, Automated tools that support the email archive pfgcess are not robust
The current version of Mail Attender is prone to failureNaming requirements are not fully supported
Archive process are not monitored with system management toolsSystem management tools can be used to atert when a process failsCompleteness of the archive process can be monitored as well
There is no "dedicated"archive tocation. Emails have been stored in variouslocations
Storage locations are created whenever theie is available space, at the directionof the operator
tlSearches of email in respoqs.e to statutory re.iquirements may-not'be complete,creating legal and political iisk
i
For Official Use Only. r! .r
.i
inab¡lity to meet statutoq¡ requirements.
o
a
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 5 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 30/75
This risk can be mitigated.:l" t.
t'l'lmplement interim procedures
ì
Daily archive v. system administrator defined periodicityStandard naming conventionsDesignated storage location(s) !
Update Mail AttenderAutomated scñeduling of the .pst creation anO sto¡åge process
Better and consistent naming supportMore reliable performance
Use Microsoft Opefation Manager (MOM) system management softruare tô monitor each pro."riand validate the integrity of the entire prooess
- Alert the messaging team and NOC when the proeess fails so that remedial action can be
taken immediatelylmplement a dedicate system that is intended for email;archive, search and record managementfunctions ,r
'i
Using vendor supported technologies to replace tools such as Find lt will improve the qualityof service .'
Systematically- eliminate manual processes and potentlal points of failure from the process!
a
For Official Use Only
¡
.,¡ t
5
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 6 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 31/75
. Release 1
lmplement proceduresUpdate Mail Attender
,ii
ttHowever, several steps. . .
o,CCDtsao9(l)v, l-o'
qllova-'É È,ôo¡E -l.È, P(ÚtoU'
E€<.=
For Official Use 9nly"il ¡t
6
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 7 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 32/75
. Release 2
lmplement system monitoring
' ;'
'f i ,¡ .
''l ',
...must be takbfì...
ctf.ECDoU'U'o
I
oðU'
co.H(Eo=o.CL
Eoot-
I
E
=Ð()r+,U'EIFc
For Official Use Only 7
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 8 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 33/75
¡
l:
¡, ¡over time.....
1., .'. ,Release3 i
lmplement dedicated email archive Ï;' ,
¡iii
CDcct)o'u,tt,o
=I
oð
U'co+ro()=o.o.
Eooþ
I
E=+,C)
=iÈ.
ît,o¡-rÞb
For Official Use Only
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 9 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 34/75
I
. Release 4 i
Upgrade Exchange servers to Exchange 2003 (for "enveloping' capability)
it ;
,..to close exposures to risk..l
Capture emails directly and continuously with ECRMS
cttc
BE
oðo
vriI-#.e9tFr. I(oEocDEÊ<=
For Official Use OnlV
:
ECRMS
I
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 10 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 35/75
Senior management authonzation is required toproceed with risk mitigation:'recommendations.
Procedures, tool upgrades and storage managei1lent decisions can be made by1
octo.l
Conversion of old emails to the ECRMS platform is time consuming. During theconversion process, the current storage volume and the ECRMS (Centerra) storagevolume will need to be searched.
- OCIO currently searches multiple volumes with Lotus Notes Mail and MSExchange email platforms.
Use of the ECRMS platform requires management authorization."Release 3" uses the ECRMS platform for a¡:,chiving. No change to search tools is
implied."Release 4" stores the email directly. Current search tools used on .pst files willneed to be replaced. However, the Find lf tool is not a vendor supported productand should bè replaced. i l
Failure to mitigate operational risks may result in en inability to comply with statutoryrequirements. 'rí
For Official Use Onty 10
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 11 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 36/75
recommendaTiorìs.
Reduces.the frequency of error typical of manudl processes.
Closes exposure to risk by isolating the email arcþive from data and file storage thatis managed to support production applications w.llere data is. more volatile.
!..'
Provides safeguards to assure the integrity of the ,process.
Systemmonitoring 'l"- Audit processes
Business rules and controls
There are s-¡gnificant benefits to the
Manages data with supported technologies.
1lFor Official Use Onfy 11
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 12 of 12
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 37/75
EXHIBIT 2
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 38/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 39/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 40/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 41/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 42/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 43/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 44/75
Case 1:07-cv-01707-HHK-JMF Document 121-3 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 45/75
Case 1:07-cv-01707-HHK-JMF Document 121-4 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 46/75
SubjecÍ Exchange MST Activity Plan
Date: Monday, November 28,2005 (Updated Friday, January 20,2006)
The following outlines the planned activities to recover Office of Vice President email ûom the target
period of September 30, 2003 to October 6,2003:
1. Environment information gathering activities
a. In the lab environme,lrt restore the backup tapes fromthe target period. This will provide technical staffwith an accurate configuration ofthesen/er and Exchange environme¡rt for the target period. (Estimate: 2 dap)
b. Perform a set ofqueries against this restored This includes identi$ing
atl registered Exchange seryers, all user objects and associated compone,nts, and all usermail stores and storage groups. This information will be used to determine which servers¡
need to be restored to recover .PST files, journal mailboxes and Exchange se,rr¡ers fromthe target period.(Estimate: I day)
The environme,lrt information gathering activities were sucoessfirlly completed. As a
result, it was determined that the Exchange sen¡erused to support the OVP mailboxes' and the Journal Mailbox server that was used &ring the target
period was OVP_JOURNAL.
Note: At this point there are three paths that will be followed:
1. Recovery of se,lr¡ers that were used to store .PST files for the target period.
2. Recovery of the Exchange servers used as the journal mailbox sewem for the target period.
3. Recovery of Exchange servers that contained the mailboxes of the componetrts in question
for the target period.
Case 1:07-cv-01707-HHK-JMF Document 121-4 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 47/75
Exchange MST Activity PlanFriday, January 20,2006
Page:2
2. Identiff and restore the serr¡ers that contain data from the target period.
a. Identi$ and restore the sen¡ers that were used to store the.PST files (estimate 3 - 5 daysper se,lner).
b. Idørtiff and restore Exchange joumal mailbox servers for appropriate component(estimate I day per seruer).
c. Ide,ntiS and restore Exchange servers for the appropriate compone,nt (estimate 5 to l0days per server).
3. Step I - Recovery of .PST files that contain message data from the gaps identified.
a. Perforrrrarlalysiso-nthe.PSTfiletodeterminethecomponents;usersand-thedata.range.i.-.ofthe messages.
b. If messages are found for the target component, users and dates, then further analysis isrequired to determine if these .PST files were included in the search activities during thetarget period.
c. If the .PST files are recovered for the entire target perio{ steps 2 and 3 may not benecessafy.
Analysis of the files contained on the file serveß that were used to store .PST files during thetargetperiod was perforrred and no messages were found that filled the gap of missingmessages for the target period.
(Draft) EOP-OA-OCIO - For Official Use Only
Case 1:07-cv-01707-HHK-JMF Document 121-4 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 48/75
Frida¡ January 20, 2006Exchange MST Activity Plan
Page:3
4. Step 2
-Recover messages from the journal mailboxes.
a. Initialize restored Exchange journal serven¡ in lab environment.
b. Configure and veriff Exchange server operation and connectivity.
c. V€riry the existence of messages in the joumal mailbox, and that the journal mailbox
contains niessages for the target users / compone,lrts for the taryetperiod.
d. If messages are found for the target period, qeate.PST files from the journal mailbox.
Fr¡rther analysis is then required to deternrine if these messages were included in the
search activities dtring ttre target period.
e. If the jounat mailbox messages a¡e recovered for the e,lrtire target period, step 3 may not
be necessarY.
Thesen.ç.r.that-containcd the jour.r¡almai,lboXes fo¡ th.e tar-get period was.suuscssfully restored,'
This was from a backup that was performed on10/2112003. The journal mailboxes were
examined and no messag€s for the target period were prese,lrt in the joumal mailbox.
5. Step 3 - Recover messages from individual user mailboxes.
a. Once the restore of the Exchange seroeß is complete, v€riry the existe,lrce of the usermailboxes.
b. Export mail messages from the att target user mailboxes. These user mailboxes should
contain all messages that existed in the mailbox at the time of the backup. This export
process would include the inbox, sent, and delete items folders as well as all personal
folders of the user.
The Exchange server that contained the OVP mailboxes for the target period was restored from
a backup that was performed onl0l2ll2003. OA Hr¡man Resources produced a list of active
OVP stafffor the target period. This list was reviewedand confirmed by OVP.
The Exchange server that contained the OVP mailboxes was restored. This was from a backup
that was performed on lO/2112003. The email from the target period was exhacted from each
of the 70 OVP mailboxes and copied to a -PST file.
(Draft) EOP-OA-OCIO - For Official Use Only
Case 1:07-cv-01707-HHK-JMF Document 121-4 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 49/75
EXHIBIT 4
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 50/75
February 21,2008
By Email
Honorable Henry A. WaxmanChairman
U.S. House of RepresentativesCommittee on Oversight and Government Reform2157 Rayburn House Office BuildingWashington, D.C. 205 l 5-6143
Dea¡ Mr. Chairman:
I am writing you in response to your request for answers to specific questions relevant to theCommittee's investigation relating to whether the Executive Office of the President has compliedwith federal laws requiring the preservation of preservation and federal records.
Ihave provided
myresponses
toyour questions
to Mr. Emmet Flood, Special Counsel to thePresident and to M. Elizabeth Medaglia, Office of Administration General Counsel. In recentweeks they have expressed to me their concerns about potential disclosures of deliberativediscussions involving the participation of Office of the Chief Information Officer management,Office of Administration General Counsel, White House Counsel's Offrce and White Housemanagement. To address these concems, I have provided my responses to them for their review.They have committed to me that they will review and identify any responses or other specificinformation that they wish to be redacted from my response prior to submission to theCommittee. Any items they choose to redact should be addressed to them.
I realize that there are many complex issues related to this topic. If you or your staff have any
additionalquestions,pl"q'"&'-!f'e.tocontactmeviaemailutto'via telephone anytime at lll]. Thank you for your time and attentiöin füffiffier.
Steven McDevitt
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 51/75
Responses from Steven McDevitt (Part I of 2)February 21,2008
General Bacþround
1. During what time period did you work at the lVhite House?
I was employed in the Office of the Chief Information Officer (OCIO) in the Office ofAdministration (OA) in the Executive Office of the President (EOP) from September
2002 through October 2006.
2. What position or positions did you hold at the t#hite House? To whom did youreport?
From September 2002 through July 2003, I was an'Information Technology Specialist -Project Manager (GS-2210-14) in the Concepts, Requirements and Systems Engineering
Directorate (CR&SE) in the OCIO. During this period I reported to Mr. Layton Clay, the
Director of CR&SE.
In July 2003,the OCIO was reorganized and the Architecture and Engineering (A&E)Directorate was created. I was promoted to the position of Director of the A&E as a
Supervisor Information Technology Specialist (GS-221 0- l5).
From July,2003 through January 2005,1 reported to Mr. Carlos Solari, the ChiefInformation Officer (CIO).
From January 2005 through May 2006,1 reported to Mr, John Straub, the Director of the
Offrce of Administration and acting CIO.
From May 2006 through the end of my tenure in the OCIO, I reported to Ms. Theresa
Pavton.
Mr. Steven McDevitt - February 21,2008 - Page I of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 52/75
3. What were your primary job responsibilities? If they changed over time, pleasedescribe your responsibilities over time.
As an Information Technology Specialist - Project Manager (GS-221 0- l4), fromSeptember 2002 through July 2003,I was responsible for managing various systemsdevelopment and systems implementation projects. During this period, the majority
ofmy efforts were focused on the implementation of a new records management system forthe White House Office of Records Management. The primary purpose of this systemwas to manage the paper records and document of the President and his staff.
During this time, I was also assign to begin the process of implementing an electronicrecords management system to manage the email and other electronic communicationsrecords throughout the EOP,
When I was promoted to the new position of Director of A&E my areas of responsibilityincreased significantly. The primary responsibilities of A&E includes:
o Systems Engineering and Integration
-Responsible for the development and
implementation of numerous custom developed applications and theimplementation of commercial-off-the-shelf (COTS) based solutions.
o Business Applications Support - Provided day-to-day management and supportfor a wide variety of applications that supported the mission of the components ofthe EOP. There were approximately sixty-five applications that support thecritical business needs of the EOP.
o Website Management and Support - The primary focus of this support was forwhitehouse.gov. This included a team of web content management staff, webdesigners and technology specialistó. Support for other websites was also
provided. Includingomb.gov, results.gov, wmd.gov and other White House
related sites.
r Enterprise Architecture - A&E was responsible for the development and
maintenance of the Enterprise Architecture (EA) of the EOP.
4. Did you have any staffwho reported to you? If so, please describe the size androle of your staff.
As Director of A&E, I had between 8 and l3 staff reporting to me. The staff was a mixof project manager, technical specialists, enterprise architect and web specialists. All
were Information Technology Specialists or Supervisory Information TechnologySpecialists (GS-2210) grades 9 through 15.
Mr. Steven McDevitt - February 21, 2008 * Page 2 of l0
This document contains sensitive network information of the Executive Office of the President,
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 53/75
5. There were varÍous contractors that worked with staff in the Office of the ChiefInformation Offtcer. tWhich contractors did you work with, and what was theirrole?
The confiactors that supported the mission of A&E included:
r Boeing - Enterprise architecture support for the development and maintenance ofthe EOP EA.
o Booz Allen Hamilton - Was awarded the contract for the implementation of theWhite House Office of Records Management, records Management system(RMS). They were also awarded the contract for the initial requirements analysisand solution selection for the Electronic Communications Records ManasementSystem (ECRMS)
e Lockheed-Martin - Support the for IntranetQuorum system used by the Office ofCorrespondence.
o MZM- Provided support for the implementation systems related to the emailinfrastrucfure.
o Systems Management and Engineering Inc. - Enterprise architecture support forthe development and maintenance of the EOP EA.
o TKC Communications - Provided systems engineering and technical assistancesupport on a wide variety of systems development and systems implementationprojects.
o Unisys - Provided systems analysis and systems implementation support. Thesewere specifrc tasks under the larger multi-year information technology support
contract that provided enterprise-wide services to the EOP. Unisys was taskedwith the implementation and integration of the ECRMS system.
Mr. Steven McDevitt - February 21,2008 - Page 3 of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 54/75
E-mail Systems and Archiving
6. The Committee undenstands that, at some point in2002, the \ilhite House began a
migration of e-mail systems, switching from Lotus Notes to a Microsoft Exchangesystem. Do you know when the decision was made to make this migration? What
was the rationale for the change? When did the migration begin and when was itcompleted?
There were multiple reasons for the desire to migrate from Notes to Exchange.
o Senior White House staff had a desire to migrate to Microsoft Outlook and
Exchange because that is what they were used to on the campaign.
o The Outlook platfoÌm was widely used in commercial enterprises and provided
better integration with the Microsoft Office suite of applications that was thestandard within the EOP.
o Also, there were a number of features of Outlook that were not available in the
Notes Mail environment.
The project to evaluate the migration to Outlook / Exchange began prior to the beginning
of my employment with the EOP in2002.
The migration for part of the Office of Administration occurred as early as September
2002. The reason I know this for certain is that when I began my employment, I was not
provided a Notes Mail account, I was provided an Outlook/Exchange account.
7. Was there any particular order dictating how the migration proceeded? Was the
migration done component by component or on a more individualbasis?
With about two thousand people to migrate from Notes to Exchange, there was a formal
process that was put in place to suppof the migration, As a general rule, the migration
was done on a component by component basis with groups of individuals migrated at a
time. The migration needed to be coordinated with the management of each component
as it impacted email of each user.
I personally had no direct operational responsibility for this process. Detailed plans were
cieated to support this migration. The OCIO should have detailed documentation on
when each user or groups of users were migrated'
Those responsible for the planning and execution of the migration included Bruce
O'Dell, the Deputy CIO during this period, Bart Hill, the Director of InformationSystems & Technology and the OCIO email support team that provided operational
support for the email systems.
Mr. Steven McDevitt - February 21, 2008 - Page 4 of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat ¡t as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 55/75
8. W¡th the Lotus Notes e-mail system, the White House used an archiving system
known as ARMS to presene e-mails sent and received by White House staff. Areyou familiar with this system? Were you aware of any concerns about the adequacyof the ARMS program? If so, please describe those concerns.
I was not involved in the implementation of the ARMS system as it was implemented in1994, prior to my employment with the EOP. My knowledge of the ARMS system was
the result of the analysis that I performed in 2002 as part of the project to implement a
long-term solution to support the email records management of the EOP.
The ARMS system is really a set of systems that were developed in 1994 to meet a court
mandated need to preserve E-mail records. At the time these systems were implemented,
no commercial-off-the-shelf (COTS) system to support email records management
existed in the marketplace.
The ARMS system was implemented using the staff, contractors, resources and
technologies that were present-at-hand within the EOP at that time. The system used
simple operating system utilities for the data management, access, search and retrieval ofdata and the frle system for the storage and access control ofthe data.
During the Clinton administration there were a number of signifïcant problems withARMS and the associated supporting systems. These problems or anomalies (Mail 2,
Letter D and Multi-Host) resulted in situations where E-mail was not appropriately
archived by ARMS. These issues were corrected and various project were completed to
recover the email that was not archived. The GAO has produced reports documenting
these issues and the resolution and conective actions that were taken.
During my analysis of the ARMS system, a number of operational and non-functional
risks and limitations were identified. These were documented in the Concept of
Operations (CONOPS) document that I createdin2002. This document was reviewed byOCIO staff, OA Records Management, OA Counsel, the White House Office of Records
Management and White House Counsel. This document was the basis for the project to
implement a COTS solution to support the email records management of the EOP.
To reduce or eliminate these risks, the ECRMS CONOPS outlined the need to implement
a system that utilized current commercially trusted technologies to support the email
records management needs of the EOP.
It ìs also important to note that by 2002 there were a number of COTS products that
provided effective email records management solution that were designed to support
seamless integration with the Microsoft Exchange platform.
Mr. Steven McDevitt - February 21,2008 - Page 5 of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential'
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 56/75
9. What was your role in planning how e-mails would be archived and preserved inthe new system? Who else was involved with this and what were their roles?
My role was to lead the effort to perform the analysis, selection a solution and implementthe solution to support the effective records management of EOP emails. This project isreferred to in various documents and presentations as ECRMS. For a period of time, theproject had the name EARS. These two names refer to the same project.
In2002, there were two other projects that This project began in late 2002 when it wasrecognized that the first two attempts to use the ARMS system to support the MicrosoftExchange environment could not be the long.term solution to support the recordsmanagement of EOP emails.
Prior to the initiation of the ECRMS project there were two attempts to continue to usethe ARMS solution.
The frrst project was an attempt to modify Windows XP and Microsoft Outlook interfaceto support integration with ARMS. There were numerous technical issues that prevented
this approach from being successful. The OCIO should have documentation on thisproject.
The second project was an attempt to use an email integration solution to manage andarchive email messages using the ARMS environment. The approach was to use LegatoEmailXtender-solution to provide a mechanism for all Outlook / Exchange E-mails to bemanaged in ARMS. The project was abandoned as the poor performance of the solutionprevented it from supporting day-to-day email message volume requirements.
I believe that Mr. Howard Sparks was responsible for both of these projects.
10. How ryere e-mails sent to and from Microsoft Exchange accounts archived andpreserued? Please describe the various steps involved and the individualsresponsible for each step, including the process through which e-mails stored injournals were saved in .pst files.
I was not directly involved in the management decision to proceed with theimplementation of Outlook / Exchange. I also did not have any operational responsibilityfor the archiving of email in either the Notes or Exchange environments.
The initial email retention process involved a manual process of copying messages fromthe Exchange journals to .pst files for storage and retention. This process was to be
performed on a regular basis.At some point, this process was partially automated using a utility designed for thispurpose. The Mail Attender utility was used to automatically copy email message fromthe journals to the .pst files on a regular basis.
The details regarding the standard operating procedures should be obtained from theIS&T Directorate within the OCIO.
Mr. Steven McDevitt - February 21,2008 - Page 6 of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 57/75
11. As the migration took place, did you have any concerns about how e-mails werebeing archived and preserved under Microsoft Exchange? What were thoseconcerns? Did you express them to anyone? When did you do this and with whomdid you share your concerns?
There was a great deal of concern about proceeding with the migration to Outlook /Exchange without having an adequate email records management solution in place. Byearly 2003, an entire year had been spent trying to identify a solution that would supportthe email records management requirements of the EOP. There were four types of riskthat were discussed on a number of occasions within the management ranks of the OCIOand OA. This risks included:
o Incomplete Data - The process by which email was being collected and retainedwas primitive and the risk that data would be lost was high. In addition to thisbeing a manual process, the risk was compounded by the fact that there was nomechanism to reconcile the messages that were retained in the .pst files and the
messages that had been processed by the Exchange system. The potential impact:The system does not contain all required data.
o Data Reconciliation - The use of .pst frles for warehousing email records does notprovide a mechanism to reconcile against what was originally retained by the-system. This is there is no way to guarantee that all records are retained in theircomplete and unmodified state. The potential impact: It cannot be demonstratedthat the data in the system is complete.
o Public Perception - Given the issues that occurred during the prior administration,it should warrant extra caution on the part of the EOP before making any changes
to the email retention process. Additional system problems would create a public
perception that the EOP was unwilling or unable retain records that were requiredunder current law. The potential impact: Increase scrutiny of the EOP andsignifïcant additional expense to correct any problems that might occur.
o User Accountability - The approach of simply storing email message in .pst filesprovides no mechanism or audit trail that tracks changes to data files or the
activities performed by users or system administrators. The integrity of the data
could be called into question because it was not possible to ensure theinappropriate action, either intentional or unintentional, could not occur. Or, ifthey did occur, the actions would be logged and the user who performed those
actions could be identified. The potential impact: No verification that data
retained has not been modified or what activities have been performed by systemusers or administrators.
Mr. Steven McDevitt - February 2l , 2008 - Page 7 of I 0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 58/75
In early 2003, prior to the large scale rollout of Exchange, these concerns we oftendiscussed within the management ranks of the OCIO and OA. People involved in thesediscussion include Tim Campen (OA Director), Carlos Solari (CIO), Bruce O'Dell(Deputy CIO), Bart Hill (IS&T Director), Jaime Borrego (Information Assurance (IA)Director) and myself.
The reason for my involvement in these discussions was that I was leading the effort toidentiff and implement the long-term email records management solution.
12. Under this Exchange system, were you aware of any avenues through which e-mail archiving could have been circumvenúed? If so, please describe those avenuesas well as any steps you or others took to prevent the loss of e-mails.
Only those email messages sent and received using the EOP Outlook / Exchange andEOP Lotus Notes environments would be included in the EOP email retention process.
Other avenues of electronic messaging included:
o Email message sent and received using non-EOP mobile devices (cell phones andPDA's) would not be retained within the EOP records.
o The use of non-EOP mobile devices to access other email service providers suchas Hotmail, Gmail or Yahoo. These messages would not be retained with theEOP records.
o Peer-to'Peer Messaging, such as PIN-to-PIN Blackbeny messages would not be
retained within the set of EOP records.
o Use of non-EOP email sites from EOP computers, such as those hosted by
political or other organizations. These records would not be retained within theset of EOP records.
The EOP Information Assurance Policy addressed each of these issues. Current OCIOemployees should be able to address these questions.
Mr. Steven McDevitt - February 21,2008 - Page 8 of l0
This document contains sensitive network information of the Executive Offìce of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 59/75
13. During your time at the lVhite House, was there every any system put in place toaudit or verify that e-maÍls were archived and preserved correctly? If so, when wasthis system put in place? Who was in charge of this audit and verification?
After the implementation of Microsoft Exchange in 2002 and2003, and after the
migrationof users from Notes to Exchange began, there was no automatic audit system
that was implemented to ensure that emails were archived and preserved.
It was also discovered in October of 2005 that there was no manual periodic accounting
or reporting process.
After the issue of potential missing email was identified in October 2005, one of the
corrective actions was to implement a standardized formal daily procedure to ensure that
the daily process to copy email messages from the Exchange journals to .pst filesoccurred without error and was completed as defined by the standard operatingprocedure.
At the time, this process was conducted on a daily basis by staff that independent of the
email operational support team. The results of this process would provide the basis of anaudit trail that could be used to validate the number, size and number of messages
retained in the inventory of .pst files.
14. Who had access to the servers that held the archived Exchange e-mails? D¡dthese servers have any extra security protections? Would these files ever be opened
or modified - for example in a search for records? Who would have had access?
Were there any protections to prevent them from being modified or deleted -either intentionally or accidentally?
I had not operational responsibility for the email retention process and I do not know theanswers to these questions.
Søff from the IS&T Directorate had operational responsibility for the EOP email systems
and the email retention processes.
In mid-2005, prior to the discovery of the potential email issues, a critical security issue
was identified and corrected. During this period it was discovered that the file servers
and the frle directories used to store the retained email .pst files were accessible by
everyone on the EOP network.
Mr. Steven McDevitt - February 21, 2008 - Page 9 of 10
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 60/75
15. Was there any policy that prevented White House staff from accessing externale-mail accounts on their official \ilhite House computers? \ilas this policy applieduniversally?
In2002 and 2003 the EOP Information Assurance Policy was drafted, reviewed and
approved, The policy was approved by each component within the EOP.The purpose of the policy was to address the wide anay of information security and
information assurance requirements of the EOP.
Relating to email, this policy specifrcally prohibited the following:
e Use of non-EOP email environments was prohibited because it would not providea means for supporting records management requirements.
o Use of encrypted email was prohibited because there \ryas no facility to manage
records retention of encrypted email.
o Use of peer-to-peer messaging was prohibited.o The use of instance messaging environments was prohibited.
Questions and information requests on the applications of this policy should be addressed
by current OCIO staff.
Mr. Steven McDevitt - February 21,2008 - Page l0 of l0
This document contains sensitive network information of the Executive Office of the President'
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 61/75
Responses from Steven McDevitt (Part 2 of 2)February 2l,2OO8
PotentÍal Losses of E-mails
16. The CommÍttee understands that, at some point in the fall of 2005, concerns arose at the\ilhÍte House that some e-mails may not have been properly archived. According to an e-
mail exchange between you and Susan Crippen at the White House, it appears that thoseconcerns may have first been raised on October ll,2005. What precipitated these
discussions about message storage issues? What was your role in these discussions, andwho else was involved?
Actually, I believe that I and some members of the OCIO management team suspected there
were issues and we discussed these issues within the OCIO management meetings a week or so
prior October 11,2005.
It was reported by the email support team to the OCIO management team that there were some
issues related to the processing of the Exchange journals and creation of .pst f,rles for each EOP
component. At the time it appeared that because of server / application reconfiguration errors
that occurred in August2005, all EOP email for most of August and September were retained as
OA email. It did not appear that any email was missing or not retained, but rather it appeared
that all EOP email was retained in a single set of OA .pst files and not the .pst tiles associated
with each component. It was also reported that they email support team attempted to take
corrective actión to conect the issues, but where unable to fix the problem and separate out the
email into their respective components.
This precipitated a series of discussion within OCIO management and staffabout how the 'pstfiles were managed and inventoried. It became clear that these files were not being effectively
managed.
Some of the issues that became known include:
o The EOP email retention .pst files were scattered across various seryers on the EOP
network.
o There was no complete inventory of all .pst files
r The processing of the Exchange joumals to create the .pst files did not always complete
during the normal processing cycle'
r There w¿$ no separation of duties or audit controls in place to ensure that the processing
of these was being performed on a consistent basis'
o Therewas no well documented process
o There was no consistently applied naming convention for the component .pst files-
o There w¿rs no daily review to ensure that all processing was completed conectly' This
point was emphasized by the fact that over a month had gone by before it was discovered
that there was a problem in August and September 2005'
Mr. Steven McDevitt - February 21,2008 - Page I of l5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 62/75
17. The White House informed Committee staff that these message storage issues mayhave been first discovered during a search for documents that found several weeks of e-mails for certain rilhite House compoüents had been stored in the wrong fÏles. Does thismatch your recollection? What was the circumstance of this search, what was found tohave been misfiled, and how was the issue resolved?
Yes this does match my recollection. During my tenure in the OCIO, I was never directlyresponsible for performing email searches. I do not know about the circumstance of thisparticular search.
I do not believe that these misfiled email messages were ever separated out back to theirassociated components. The current OCIO staffshould be able to answer this question.
18. At some point, the White House âppears to have expanded its search for misfÏled e-
mails. At this point, the Committee staff understands that you undertook an analysis of e-mails preserved from the Microsoft Exchange environment breaking down the analysis by
each of 12 components of the White House. When and why did this process begin? Whoworked with you on the process? How did you conduct the analysis?
The process originated when it became apparent in October 2005 that OCIO staff and contractorswere not effectively managing the .pst files used to retain the email records for the EOP. This setof issues was brought to the fore by the .pst f,rle management problems that occurred in Augustand September of that year.
The initial set of actions was simply to organize and inventory the .pst files used for EOP emailrecords retention and to put in place a formal process to manage these files. The primary issuewas the .pst files were scattered across various servers on the EOP network. To the best of myrecollection, these series of events included:
o Performed a search all servers on the EOP network for all .pst files to identiÛ and locateall .pst files in the EOP environment
o Collect a data set that contains all relevant information about these frles (Name, location,size, creation date, etc.)
o Create a secure and organized server environment in which these files could be stored.
o Copy all .pst files to this new secure and organized location
¡ Veriff and validate that all actions to copy these files completed successfully.
o Create an inventory of all .pst flrles and verify all the information.These activities were performed by a team of OCIO staff and contractors. Each step of theprocess was discussed and documented. The team met on a daily basis to plan activities and toreport on actions that had been completed.
Mr. Steven McDevitt - February 21,2008 - Page 2 of l5
This document contains sensitive network information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 63/75
In addition to this .pst file analysis, the team also began the development formal dailyverification process that would support the effective management of these files and tire processthat created them.
While this process was taking place, I began to notice a few anomalies with these files. Theseincluded:
o .pst files that contained no data. The file size was zero b¡es.
o Inconsistent naming of files that made it difficult to determine the associated componentand date to which the file was associated.
o Obvious gaps in the date ranges represented in filenames of the file. As an example, onefrle may have been named "OA May l-5" and another file "OA May 8-10" but thereappears to be no file that represented May 6 and 7 . This is just an illustrative example.
o There was a wide disparity of frequency of how often .pst files were created for eachcomponent.
o There was a wide disparify in size of files that represented similar periods of time.Because of these issues and because there was no way to effectively determine what data wasretained in each file, the team took on the task of performing an additional level of analysis.
If my recollection is correct, at that time there were over 5,000 .pst files with an average size ofapproximately 2 Gigabytes. Since each of these files contained messages one or more days andsince it was not possible to determine what days were included in any given file, we needed todetermine a method to perform this analysis. Prior to this effort, Microsoft had provided theEOP with a custom software application for performing searches on .pst files. This tool wascommonly referred to as the "Findlt" tool.
Microsoft was contactedand was tasked to modify the Findlt tool so that it included theadditional functionality of providing a message count for each day represented in a given .pst
file. This process was performed on each .pst file in the inventory and the data was aggregatedinto a single data set. This is the data set that provided the basis for the analysis.
It took a couple week to perform the analysis on the thousands of .pst files. V/hen the data wastabulated it became clear that a problem existed because there were days for which no email wasretained. Extensive testing was performed at that time to ensure that the tools and the tabulationprocessed was performed correctly. An independent verification and validation was alsoperform by a different set of contractors to ensure that this analysis process was completedcorrectly and that the data was coffectly analyzed and the accurately represented.
In addition to there being hundreds of days for which specific components had no email retained,there were a number of days for which it was clear that the number of emails retained was lowerthan expected.
There was a formal analysis to determine if the number of days for which the number of retainedEOP emails was lower than what one would expect based on the email volume trends. Theanalysis determined that there was a clean pattem of email volumes. This analysis accounted for
Mr. Steven McDevitt - February 21,2008 - Page 3 of 15
This document contains sensitive nefwork information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 64/75
working days and non-working days (weekends and federal holidays). A multi-week movingaverage model was employed in various version of the analysis to account for normalfluctuations of normal email volume. Depending on the assumptions made in the analysis, theteam identiñed hundreds of days for which the volume of email was inexplicably low.
Those who worked on this team are listed in response to question 19.
19. The White House provided the Committee with t23 ptge color chart titled,,rEOPExchange Environment - All Components," with the subtitle, ó'SummarT
- Messages perI)ay." It lists the total days with zero messâges and low messages for 12 components of the\ilhite House, as well as the actual message count for each of these components for theperiod from Januaty 1,2003 to August 10,2005. The copy provided to the Committee isdated F'eb. 6, 2006, at 4:13 p.m. \üere you involved in the creation of this chart? If so,what was your role? Who else was involved in the development and production of thechart?
I was responsible for leading the team that created this chart. The chart to which you refer was
the result of many weeks of analysis that involved over a dozen people. I was responsible fordesigning the chart and had a leading role in the definition and execution of the analysis.
To the best of my recollection, those involved in this effort, in addition to myself, included:
EOP Employees - Jaime Borrego (Acting IS&T Director), William Reynolds (Deputy Director,Information Assurance), Vic Bernson (OA General Counsel), Keith Roberts (Deputy OAcounsel), Howard sparks (IS&T), sue crippen (IS&T), Bryan Reese (ls&T), stephenWarshauer (IS&T), Keith Regatts (A&E), Aimee Felker (Director OA Records Management),Shaffers Rawlings (EOP Records Management)
Contractors -I|I (Unisys), (Unisys), ff (SRA) and various
contractors whose names I cannot recall from Microsoft and SRA.
20. Was this the final version of the chart? If not, when was the last version of the chartcreated?
I reviewed the chartprovided to the Committee and I am not able to determine if the versionprovided is the final version. There were many version of this analysis. Each version wasidentified with a unique version number. Different version of the analysis included differentassumption about date ranges and thresholds.
I do not recall the exact number of versions of this analysis, but I believe it was between 12 and
20. What can be said is that what was provided to the Committee is just the analysis summaryreport, not the complete analysis.
The complete analysis was approximately 250 pages in length. It included the completebackground data and trend analysis.
Mr. Steven McDevitt - February 21,2008 - Page 4 of l5
This document contains sensitive network information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 65/75
21. Please describe the steps you and others took to perform the analysis required for theproduction of this chart. What types of files did you search? Where did you look for thesefiles? Did you face any challenges related to files being misnamed, too large, corrupted, orhaving other such problems?
In addition to the response to question 18.
During the process to organize and inventory the .pst files, there were a small number of filesthat appeared to be comrpted. Additional analysis was performed on these files. I do not recallthe specific outcome of these analysis, but the data in there files were not for the periods forwhich data was missing.
Z2.Each of the 12 components has a different start date on the chart and a different enddate. Can you explain why this is?
Each component has a different start date because components were migrated from Notes toExchange over a several
monthperiod.
The OCIO should have detailed list of users and the schedule of when users and componentswere migrated.
23. Was this chart the only result of your analysÍs of messages from the EOP Exchangeenvironment? If not, did you produce any other briefing maferials or documents thatexplained your methodology or findings? \ilhat were those documents?
There were numerous documentso PowerPoint presentations and other memoranda that describedthe analysis that was performed, the actions taken to correct the process and the
recommendations to improve the processing of .pst files. The team documented the details ofeach action taken to clean up and correct the identified issues.
There must be thousands of email messages between the team members that describe the actionsof the team, the completion of specific tasks, analysis of issues and to provide status to OCIOmanagement, OA Counsel and OA management.
Mr. Steven McDevitt - February 2l,2008 - Page 5 of l5
This document contains sensitive network information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 66/75
z4.Didyou or anyone with whom you worked, ever estimate the number of e-mails thatmÍght be missing? What was that estimate?
Yes, there were a series of estimates based on various assumptions. On the low end, there were
about 470 days (as reported by the Committee) days with missing email for individuals
component in the EOP. If other assumptions were taken into account, that number increased to
orr.i1,000 days of missing email. This is because of days for which Exchange was used prior toJanuary 2003 and some issues that may be been associated with the emails retained during the
August and September 2005 timeframe.
I do not recall the exact number of estimated missing email, but I believe it was greater than
1,000,000. This estimate did not include the days for which the number of retained emails was a
statistically low number compared to the predicted number of email that should have been
retained for a particular day.
The statistical prediction was based on a six-week moving average that separated working days
from non-working days. Non-working days included weekend and federal holidays and days
that traditionally have a low number of messages. An example of this type of day is the Friday
after Thanksgiving. The use of a six week moving average also accounted for the naturalseasonal fluciuations in email volume. An example of low volume are weeks during the month
of August when the President is on vacation. Also, using this moving average allowed for
accounting for spikes in email volume as a result of world events.
25. Did anyone veriff the findings of your analysis? If so, who performed that verification
and when did it occur? \ilhat did the verification find?
During this analysis process, a high level of formality and review was performed on every step
of the procerr. Íh. team performing the analysis met daily and in some cases multiple times per
day. Éach activity was documented either in meeting notes or in emails distributed to the entireteam.
As stated in a previous response, an independent verification and validation was performed by a
separate set of contractors who were not members of the team that was perforrning the analysis
effort.
Mr. Steven McDevitt - February 21,2008 - Page 6 of l5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential'
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 67/75
26,Towhom did you present the results of your analysis? When did the presentation orpresentations occur? Please describe in general what you told these individuals in these
briefings and what materials you used.
The results of this analysis, both in preliminary and final form were presented on various
occasions to OA management and OA counsel. I do not have any specific dates on when these
meetings occurred, but they did occur often throughout the period of October 2005 throughFebruary 20A6.
Outside of OA, there were other meetings with the purpose of presenting these issues to White
House Management and White House Counsel and their staff.
The summary chan or other charts similar to it were used in the discussion along with other
PowerPoint presentation that presented the critical facts and recommendations. The briefing
described the problem and presented various option to prevent the issues from recurring and to
qorrect the problems that had occurred.
27.The Committee understands that you and John Straub met with White House Counsel
Harriet Miers to discuss issues related to e-mail preservation. Did you discuss youranalysis at this meeting? Please describe when this meeting occurred, the agenda for the
meeting, and your recollection of what was discussed.
I participated in a number of meetings in December 2005 and January and February 2006. Some
oithese meetings included White House Counsel Harriet Miers and members of her staff. These
meetings also included other White House management and OA Counsel staff.
Given the nature of these discussions, I will defer to the current White House staff to
characterize these meetings.
28. To your knowledge, were other presentations of the findings made when you were not
in attendance? Who made those presentations? To whom did that person present the
findings?
I would assume that other presentations of the findings of the email analysis team were made
when I was not present. However, I cannot recall any specific details about any such
presentations.
Mr. Steven McDevitt - February 21,2008 - Page 7 of 15
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 68/75
29.lVas any analysis conducted to determine how e-mail files could have been lost? If so,
who took the lead on that, and what was the determination?
From October 2005 through August 2006, no analysis was conducted to determine the cause ofthese issues. No direction was given by OA management to undertake such an analysis.
The primary focus of the team addressing these issues was to create a proper inventory of ,pstf,rles, analyze these files to determine if issues related to email retention existed.
30, Did the White House ever inform the National Archives of the results of your analysis?If so, when was this done? If not, did you or any others recommend that this be done?
During my employment with the EOP, I do not recall if anyone at NARA was informed aboutthese issues.
Sometime during the Summer of 2006,1 was directed by the CIO that I was not allowed to
discuss the potential email retention issues and the analysis that was performed by OCIO withthe NARA staff. I was to inform any NARA staff who contacted me about these issues to directall inquires about email records management to White House Counsel and White House Records
Management.
During my employment at the EOP, I worked closely with NARA staff on a number of issues
related to records management. I had established good working relationships with them. Ireceived a number of inquiries from them and in each case I redirected their inquires to the
White House. I was verv clear to them that I was directed not to share information with them.
Efforts to Develop New Archiving Systems
31, It appears that the White House took several steps toward developing a new system forarchiving and managing e-mails during the Bush Administration, including a contract forLegato to build an Exchange Interface System that would enable the ARMS system toarchive and manage Exchange e-mails. Were you involved in or aware of this contract? Ifso, what was your role in the preparation and oversight of the contract?
I was not directly involved in the contract or the management of the work associated with the
creation of the Exchange Interface using the Legato EmailXtender product. This project was
managed by Mr. Howard Sparks. I was involved in the evaluation of the solution that was
implemented. Once the solution was created and was being tested it became clear that the
system would not meet the performance requirements necessary to support the daily volume of
email processed by the EOP. In early 2003, this project was abandoned.
Mr. Steven McDevitt - February 21,2A08 - Page 8 of I 5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 69/75
32. What do you underutand to be the results of this contract? Was the Exchange-ARMSinterface ever implemented? Why or why not?
It is my recollection that the interface w¿rs never fully implemented. The testing of the initialcapability indicated that it would not meet the performance requirements necessary to supportthe normal daily email volume of the EOP.
The failure of this approach (the use of EmailXtender to move email message into ARMS)created the situation where if the migration from Notes to Exchange were to proceed, therewould be not automatic email records management functionality. In spite of this situation, WhiteHouse and OA management made the decision to proceed with the migration.
In order to meet basic records management requirements, White House and OA managementalso made the decision to retain all email messages processed through the EOP Exchangeenvironment, using the Exchange joumaling capability and copying rnessage to .pst files forstorage.
33.It also appears that the \ilhite House took steps to develop a new electronic archivingsystem known as the Electronic Communications Records Management System, orECRMS. Were you involved in or aware of this contract? If so, what was your role?
34. When was the concept for ECRMS first developed? Who led the effort to plan for anddesign the system? If contractors were involved, please describe which contractors workedon the effort, when they became involved, and what their role was.
35. It also appears that the Office of Administration prepared a Statement of Work for anE-mail ArchÍve Retrieval System. A draft Statement of Work, dated September 2112004,notes that, ú'the primary goal of this engagement is to provide EOP staff with a solution
that allows them to archive, manage, search and retrieve E-mail they may want to storeand preserve on a long-term basis." EARS is described as being related to ECRMS. Youare listed as the author of this draft. Was a contract Íssued for EARS? If so, to whom wasit granted, what was the time frame of the contract, and was the project implemented?
[Combined response to questions 33, 34 and 35]
It was recognized by OA and CIO managementin2}0? that the EOP needed a long termsolution for email records management. I was assigned the initial project management role in2002. lnJuly 2003, the project was managed by various staff in the A&E Directorate.
Because of the issues related to email records management that had occurred in the past, a highlevel of scrutiny and caution was applied to this project. This involved additional periods ofreview of various work products. These reviews include White House and OA Counsel, WhiteHouse Office of Records Management and OA Records Management.
Mr. Steven McDevitt - February 21,2008 - Page 9 of l5
This document contains sensitive network information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 70/75
My recollection of some of the specific dates may be off. The rough chronology for the ECRMSimplementation is as follows:
o November - December 2002 - The initial draft of the Concept of Operations (CONOPS)for the ECRMS system was completed.
o December 2002 through May 2003 - The ECRMS CONOPS was reviewed and approvedby OA Counsel, White House Offlrce of records Management and White House Counsel.
o April lMay 2003 - The Statement of Work for the initial phase of the project was draftedand the procurement process began. The scope of this effort was to complete a detailedsystems requirements specification, evaluate commercial-off-the-shelf products andpropose solutions that meet the government requirements. The government would thenselect the solution that provided best fit to the EOP environment and the contractor wouldcomplete and delivery the design for the implementation of that solution.
o , September 2003 - Vendor proposals were received and evaluated and a selection wasmade. Booz Allen Hamilton was awarded the contract for Phase I of ECRMS.
o November / December 2003 - Initial phase of the ECRMS project began.
o December 2003 - May(?) 20A4 - Requirements analysis was completed by thecontractor. COTS solutions were evaluated against those requirements. Arecommendation was made by the vendor and the Govemment selected a the solution.The design for the implementation of the solution was created by the contract anddelivered. The solution selected was a combination of two COTS products, MDYFileSurf and KVS Enterprise Vault
o April - May(?) 2004 - The solution design was presented to OA Counsel, White HouseRecords Management and White House Counsel for their concurrence.
o June - September(2)2004 - ECRMS Phase 2,the systems implementation began. It was
decided that the current Unisys contract could be used to support the installation and
configuration of the system. The procurement and installation of the hardware and
software also occurred during this time.
o November 2,2004 - Final configuration and completion of the File Plan used to archiverecords was drafted.
o January 2005 - October 2005 - System configuration, testing and tuning. Testing withlarge email volumes to ensure that system performance would satisfy the requirements.
A number of issues were identified and the vendors conected and testing continued.
o October 2005 - February 2006 - ECRMS project impacted as the contractor staff was
supporting the clean up of the .pst file issues.
e January 20A6 - March 2006 - Large volume testing continued. The ECRMS standardoperating procedures were drafted and provided to staff for review and comment. Largescale testing was being performed using .pst files that contained OA email messages.
Mr. Steven McDevitt - February 21,2008 - Page l0 of l5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 71/75
March 2006 - White House Counsel, White House Office of Records Management andOA Counsel provided a detailed briefing on the solution.
July2007 - White House Counsel, White House Offìce of Records Management and OACounsel provided a detailed briefing on the search and retrieval capabilities of theECRMS solution.
August 21,2006 - ECRMS ready to go live.
36. Was ECRMS ever implemented? What was the status of ECRMS when you left the\ilhite llouse? Was there eyer a pilot program to test the ECRMS system? What is yourunderctanding of why it was not implemented? Who made the decision not to implementthis system?
The ECRMS system was ready to be implemented. The hardware and software was procured,installed and configured. Extensive testing was performed in 2005 and 2006. This testing wasperformed to ensure that the system would work as designed and to ensure that it would support
the performance requirernents necessary to support the daily volume of EOP email.
When I Ieft the OCIO in August 2006, the system was ready to be used. The only remainingtasks were to obtain approval from the OA Director and the CIO.
37. The White House also provided the Committee with documents indicating that yourecommended changes to the e-mail archival process in November 2005. In what appearsto be a draft memorandum from you to John Straub, dated November 14,2005, yourecommend the adoption of Standard Operating Procedures for e-mail archiving as well ast6system monitoring of the archive process" in the interim. You also recommend a "long
term risk mitigation plan" involvíng the adoption of ECRMS. Did you provide such amemorandum to John Straub? Why did you make these recommendations at this time?'Were your recommendations approved? What is your understanding of why theserecommendations rvere or were not approved?
It is my recollection that these recommendation were provided to the OA Director. I made these
recommendation because of the various situations that had occurred over the prior three years
with respect to the management of EOP email records. I felt that the procedures needed to be
formalized and I also felt that if the appropriate resources were applied to the implementation ofECRMS, the project could be completed in a timely manner and could provide a good solutionthat would prevent the reoccurrence of these issues.
I do not recall the final disposition of these recommendations.
Mr. Steven McDevitt - February 2l, 2008 - Page I I of l5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 72/75
38. A briefing document from October 2005 discusses the "operational risk in currentemail storâge management processes." This document expresses concern that the currente-mail management systems may not meet the statutory requirements for documentpreserratÍon and proposes several "risk mitigation" steps. Were you involved in thecreation of this document? If so, please describe the purpose of the document, the
individual or individuals who prepared it, and to whom it was presented.If my recollection is correct, I was involved in the creation of this presentation but I do not recallthe specific purpose of this presentation.
I do not recall to whom it was presented.
39. How was e-mail archived and preserved when you left the White House?
When I left my employment with the EOP, the semi-automated process of daily processing ofExchange journal files using the Mail Attender utility was being performed. The daily manual
process of validating that the .pst creation process completed successfully was being performedand the status was being reported to the OA Records Manager on a weekly(?) basis.
40. What was the role of the National Archives in the process of planning for anddeveloping new systems for archiving e-mail? Did officials with the OffÏce ofAdministration or the EOP consult with officials at the Archives?
The stafffrom the National Archives and Records Administration CNARA) were briefed a
number of times in2003,2004 and 2005 about the approach for email records management.
Searches of E-mails in Response to Investigative Requests
4L. Were you ever involved in, or aware of, searches for e-mails in response to investigativerequests? When? What was the subject matter? The most well known investigativerequest was for documents relating to the leak of the identity of CIA agent Valerie Plame.IVere you involved Ín a search for documents responsive to this request? If so, what wasyour role?
During my tenure at the EOP, I did not have any operational responsibility for the performanceof email searches.
Mr. Steven McDevitt - February 21,2008 - Page l2 of l5
This document contains sensitive network information of the Executive Office of the President.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 73/75
42.How did searches vary for e-mails saved in thc ARMS system and for those sent andreceived through Exchange? For e-mails saved on Exchange, did the Office of the CIOsearch files preserved on servers, on disaster recovery tapes, or elsewhere? What was thebasis for that decision?
During my tenure at the EOP, I did not have any operational responsibility for the performance
of email searches.
The OCIO has specific search procedures that describe how searches are to be performed in boththe ARMS environment and using the .pst frles used for email retention. In both cases, theprimary storage of these data set in on file servers in the EoP environment.
Searches are perform against these files.
43. Do you recall any concerns that the searches were not picking up all of the responsive e-mails? Or did you hear that the searches ever reyealed erors in the way e-mails werepreserved? Ifso, how
didyou
respond?The use of primitive search tools, both in the ARMS search and the search of the .pst files, wasraised on a number of occasions. The tools that were used were both slow and primitivecompared to current off-the-shelf search technologies.
Each time a search was performed it consumed an enorrnous number of staff and contractorresources to set-up and perform the search.
The fact that both the ARMS and .pst file search processes did not search the email attachmentswas raised on a number of occasions. At the time I believed that this was a short-lived problemas the ECRMS solution would provided fast and effective full search capabilities, including thesearch of attachments.
Mr. Steven McDevitt - February 21,2008 - Page l3 of l5
This document contains sensitive network information of the Executive Office of the Presidenf.Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 74/75
44. At some point in late 2005 or early 2006, Ít appears that the White House alertedSpecial Prosecutor Patrick Fitzgerald that some e-mails may not have been provided to theprosecutor in response to his investigation. According to a January 23,2006,letter fromMr. Fifzgerald to aftorney representing former aide to the Vice President, Lewis Libby:o'fn an abundance of caution, we advise you that we have learned that not all email of the
Office of the Vice President and the Executive Office of the President for certain timeperiods in 2003 was preserred through the normal archiving process on the White Housecomputer system." According to court filings, the White House produced 250 pages of e-
mails from the Office of Vice President to the Special Prosecutor in February 2006. Whenwcre you first made aware that not all e-mail responsive to the Special Prosecutor'sinvestigation was preserved through the normal archiving process? Why weren't thesepâges included in the original document production? How were you made awåre of this?\ilhat steps did the White House take to restore these e-mails? Where did these 250 pages
of e-mails come from?
Dwing the period in October through December 2005, when the .pst file organization and
analysis was occurring, it became known that some of the periods for which not email was
present in the retained .pst files were the same periods for which Special Prosecutor PatrickFitzgerald had subpoenaed the White House for emails related to his investigation.
Most critical were a set of days in early October 2003 where it appears that all email for the
Office of the Vice President was missing. A detailed plan was developed to attempt to recover
the email for this period.
This plan was prepared by the OCIO staff and presented to White House Counsel. I do not recallthe specific details of this plan. A number of the activities identified in the plan were undertaken
and to the best of my recollection, the email from the period in question was never recovered.
I worked with OA Counsel and White House Counsel on efforts to provide an explanation to the
Special Prosecutor. This included providing a briefing to the Special Prosecutor's staff on thissubject.
There was a parallel effort to attempt to recover all ernail from this period. The results of thiseffort were the 250 pages of email. However, I was not directly involved in this process and am
unable to provide any details relating to the 250 pages of email.
Mr. Steven McDevitt - February 2l , 2008 - Page 14 of I 5
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P
8/7/2019 CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition to Defendants Motion to Dismiss
http://slidepdf.com/reader/full/crew-v-eop-regarding-missing-wh-emails-22009-crews-opposition-to-defendants 75/75
45. The Committee was provided with an e-mail exchange between you and Susan Crippen,with copies to Jaime Borrego and Wiltiam Reynolds that attached an *Exchange MSTAcúivity Plan" dated November 28,2005 and updated on JanuarA 20,2006. The attachedplan states: "The following outlines the planned activities to recover Oflice of VicePresident e-mail from the target period of September 30, 2003 to October 6, 2003." Was
this search relevant to the Special Prosecutor's investigation? Why was this periodtargeted? What was the role of each of the individuals on this e-mail with regard to theactivity plan?
Yes, the attempt to recover these email was in response to the search associated with the SpecialProsecutor' s investigation.
The period was targeted because it was among the set of date which were of interest to theSpecial Prosecutor.
The individuals involved represented the OCIO management staffthat was in place at that time.Susan Crippen was the Deputy Director of Information Systems and Technology, Jaime Borregowas the acting Director of Information Systems and Technology and the Director of InformationAssurance, William Reynolds was the Deputy Director of Information Assurance and I was theDirector of Architecture and Engineering. All the individuals identified were involved in boththe activities to correct the .pst file management problems and with the activities associated withattempts to recover missing emails to support the response to the subpoena from the SpecialProsecutor.
Disaster Recovery Tapes
46. According to the White House, until October 2003, the EOP disaster recovery back-uptapes were recycled and were not preserved. Were you involved in this decision? To your
knowledge, who was involved in this decision? Did anyone ever express any concerns toyou about the decisÍon to recycle all of the tapes? If so, what were the concerns and whoexpressed them to you? Did you have any concems about this recycling of tapes? If so,please explain.
During my tenure at the EOP, I did not have any operational responsibility for the managementof backup tapes nor was I involved in decisions related to the recycling of backup tapes.
47.The White House has also told Committee staff that thÍs recycling was temporarilystopped several times - in February 2002, July 2002, and September 2003 before it was
Case 1:07-cv-01707-HHK-JMF Document 121-5 Filed 02/20/2009 P