OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal...
Transcript of OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal...
CBP is on Track to Meet ACE Milestones but It Needs to Enhance Internal Controls
May 11 2015 OIG-15-91
DHS OIG HIGHLIGHTS CBP is on Track to Meet ACE Milestones but It Needs
to Enhance Internal Controls
May 11 2015
Why We Did This The Automated Commercial Environment (ACE) is the commercial trade system designed to automate border processing to enhance border security and foster our Nationrsquos economic security ACE is part of a multi-year US Customs and Border Protection (CBP) modernization effort that is being deployed in phases and must be completed by December 2016 We conducted the audit to determine whether CBP is on track to meet its milestones for the implementation of ACE
What We Recommend We made one recommendation to continuously assess evaluate and update internal controls to include a risk assessment that identifies potential data reliability gaps and develop and implement specific measurable achievable relevant and time-sensitive performance measures This recommendation when implemented should improve the efficiency and effectiveness of the program
For Further Information
Contact our Office of Public Affairs at (202) 254-4100 or email us at DHS-OIGOfficePublicAffairsoigdhsgov
What We Found CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured to a rapid deployment strategy called Agile Currently in large part due to the implementation of Agile CBP is on track to meet its milestones for the deployment of ACE However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program Similar internal control issues were identified in a 2014 KPMG financial statement audit If these weaknesses continue and internal controls do not keep pace with Agile and the rapid implementations of the ACE program deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBP Response In its response to our draft report CBP reported that it appreciated the Office of Inspector Generalrsquos (OIG) recognition that the ACE program is on track to meet its implementation milestones and OIGrsquos positive conclusion that the Agile development methodology improves the effectiveness and efficiency of the ACE development process However CBP did not concur with our report recommendation
wwwoigdhsgov OIG-15-91
vttllf r
~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security
Washington DC 20528 wwwoigdhsgov
MAY 11 2015
MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection
FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits
SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls
For your action is our final report CBP is on Track to Meet ACE Milestones but
It Needs to Enhance Internal Controls We incorporated the formal comments
provided by your office
The report contains one recommendation aimed at improving the Automated
Commercial Environment program Your office did not concur with the
recommendation Based on information provided in your response to the draft
report we consider the recommendation open and unresolved As prescribed
by the Department of Homeland Security Directive 077-01 Follow-Up and
Resolutions for the Office of Inspector General Report Recommendations within
90 days of the date of this memorandum please provide our office with a
written response that includes your (1) agreement or disagreement
(2) corrective action plan and (3) target completion date for the
recommendation Also please include responsible parties and any other
supporting documentation necessary to inform us about the current status of
the recommendation Until your response is received and evaluated the
recommendation will be considered open and unresolved
Consistent with our responsibility under the Inspector General Act we will
provide copies of our report to congressional committees with oversight and
appropriation responsibility over the Department of Homeland Security We will
post the report on our website for public dissemination
Please call me with any questions or your staff may contact Paul Wood Acting
Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also
send your response to OIGAuditsFollowup(c~oi~ dhsbullgov
Attachment
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo
Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage
The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status
In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE
wwwoigdhsgov 2 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million
Results of Audit
CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues
If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBPrsquos ACE Program on Track
CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams
CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which
1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion
wwwoigdhsgov 3 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
DHS OIG HIGHLIGHTS CBP is on Track to Meet ACE Milestones but It Needs
to Enhance Internal Controls
May 11 2015
Why We Did This The Automated Commercial Environment (ACE) is the commercial trade system designed to automate border processing to enhance border security and foster our Nationrsquos economic security ACE is part of a multi-year US Customs and Border Protection (CBP) modernization effort that is being deployed in phases and must be completed by December 2016 We conducted the audit to determine whether CBP is on track to meet its milestones for the implementation of ACE
What We Recommend We made one recommendation to continuously assess evaluate and update internal controls to include a risk assessment that identifies potential data reliability gaps and develop and implement specific measurable achievable relevant and time-sensitive performance measures This recommendation when implemented should improve the efficiency and effectiveness of the program
For Further Information
Contact our Office of Public Affairs at (202) 254-4100 or email us at DHS-OIGOfficePublicAffairsoigdhsgov
What We Found CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured to a rapid deployment strategy called Agile Currently in large part due to the implementation of Agile CBP is on track to meet its milestones for the deployment of ACE However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program Similar internal control issues were identified in a 2014 KPMG financial statement audit If these weaknesses continue and internal controls do not keep pace with Agile and the rapid implementations of the ACE program deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBP Response In its response to our draft report CBP reported that it appreciated the Office of Inspector Generalrsquos (OIG) recognition that the ACE program is on track to meet its implementation milestones and OIGrsquos positive conclusion that the Agile development methodology improves the effectiveness and efficiency of the ACE development process However CBP did not concur with our report recommendation
wwwoigdhsgov OIG-15-91
vttllf r
~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security
Washington DC 20528 wwwoigdhsgov
MAY 11 2015
MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection
FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits
SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls
For your action is our final report CBP is on Track to Meet ACE Milestones but
It Needs to Enhance Internal Controls We incorporated the formal comments
provided by your office
The report contains one recommendation aimed at improving the Automated
Commercial Environment program Your office did not concur with the
recommendation Based on information provided in your response to the draft
report we consider the recommendation open and unresolved As prescribed
by the Department of Homeland Security Directive 077-01 Follow-Up and
Resolutions for the Office of Inspector General Report Recommendations within
90 days of the date of this memorandum please provide our office with a
written response that includes your (1) agreement or disagreement
(2) corrective action plan and (3) target completion date for the
recommendation Also please include responsible parties and any other
supporting documentation necessary to inform us about the current status of
the recommendation Until your response is received and evaluated the
recommendation will be considered open and unresolved
Consistent with our responsibility under the Inspector General Act we will
provide copies of our report to congressional committees with oversight and
appropriation responsibility over the Department of Homeland Security We will
post the report on our website for public dissemination
Please call me with any questions or your staff may contact Paul Wood Acting
Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also
send your response to OIGAuditsFollowup(c~oi~ dhsbullgov
Attachment
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo
Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage
The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status
In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE
wwwoigdhsgov 2 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million
Results of Audit
CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues
If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBPrsquos ACE Program on Track
CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams
CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which
1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion
wwwoigdhsgov 3 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
vttllf r
~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security
Washington DC 20528 wwwoigdhsgov
MAY 11 2015
MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection
FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits
SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls
For your action is our final report CBP is on Track to Meet ACE Milestones but
It Needs to Enhance Internal Controls We incorporated the formal comments
provided by your office
The report contains one recommendation aimed at improving the Automated
Commercial Environment program Your office did not concur with the
recommendation Based on information provided in your response to the draft
report we consider the recommendation open and unresolved As prescribed
by the Department of Homeland Security Directive 077-01 Follow-Up and
Resolutions for the Office of Inspector General Report Recommendations within
90 days of the date of this memorandum please provide our office with a
written response that includes your (1) agreement or disagreement
(2) corrective action plan and (3) target completion date for the
recommendation Also please include responsible parties and any other
supporting documentation necessary to inform us about the current status of
the recommendation Until your response is received and evaluated the
recommendation will be considered open and unresolved
Consistent with our responsibility under the Inspector General Act we will
provide copies of our report to congressional committees with oversight and
appropriation responsibility over the Department of Homeland Security We will
post the report on our website for public dissemination
Please call me with any questions or your staff may contact Paul Wood Acting
Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also
send your response to OIGAuditsFollowup(c~oi~ dhsbullgov
Attachment
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo
Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage
The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status
In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE
wwwoigdhsgov 2 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million
Results of Audit
CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues
If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBPrsquos ACE Program on Track
CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams
CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which
1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion
wwwoigdhsgov 3 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo
Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage
The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status
In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE
wwwoigdhsgov 2 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million
Results of Audit
CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues
If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBPrsquos ACE Program on Track
CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams
CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which
1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion
wwwoigdhsgov 3 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million
Results of Audit
CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues
If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE
CBPrsquos ACE Program on Track
CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams
CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which
1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion
wwwoigdhsgov 3 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances
According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)
Risk Assessment Needed
CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is
1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are
being achieved
As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system
During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without
wwwoigdhsgov 4 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE
Performance Measure Updates Needed
ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program
CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review
Recommendations
Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically
a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and
time-sensitive (SMART) performance measures
wwwoigdhsgov 5 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Management Comments and OIG Analysis
In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report
Management Comments
CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls
OIG Analysis
In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment
CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors
wwwoigdhsgov 6 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps
In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved
In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures
Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation
Objective Scope and Methodology
The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department
We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations
wwwoigdhsgov 7 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014
We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service
We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives
We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions
Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer
wwwoigdhsgov 8 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix A Management Comments to the Draft Report
wwwoigdhsgov 9 OIG-15-91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
wwwoigdhsgov 10 OIG-15-91
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFF
ICE
OF
INSP
EC
TOR
GEN
ER
AL
Dep
artm
ent
of H
omel
and
Sec
uri
ty
App
endi
x B
A
CE
Pro
gram
Tim
elin
e
Oct
ober
201
0 O
ctob
er 2
011
Apr
il 20
01
June
201
3 Ja
nuar
y 20
15
Janu
ary
2016
D
ecem
ber
2016
CB
P in
itia
tes
DH
S A
cqu
isit
ion
O
rigi
nal
AC
Ere
visi
on o
f AC
E
Rev
iew
Boa
rd
CB
P ob
tain
s D
eplo
ymen
t D
Dep
loym
ent F
Man
date
d D
evel
opm
ent
plan
nin
gde
cisi
on to
ap
prov
al fo
r
expe
cted
expe
cted
to
bede
adlin
e fo
r A
CE
be
gin
s do
cum
enta
tion
pr
ocee
d A
gile
to
be
com
plet
ed
com
plet
ed
com
plet
ion
June
200
9 D
ecem
ber
2010
Ja
nuar
y 20
13
May
201
4 Ju
ly 2
015
July
201
6
Prog
ram
C
BP
star
ts
Dep
loym
ents
C
BP
appr
oves
Dep
loym
ent E
Dep
loym
ent
G
Ass
essm
ent
Agi
le p
ilot f
orre
vise
d ac
quis
itio
n
AB
an
d C
had
ex
pect
ed t
o be
expe
cted
to
beD
esig
n R
evie
wA
CE
st
rate
gy
been
rel
ease
d by
com
plet
ed
com
plet
ed
Team
th
is d
ate
reco
mm
ends
A
CE
not
co
nti
nu
e
Sour
ce C
reat
ed b
y D
HS
OIG
11
ww
wo
igd
hsg
ov
OIG
-15-
91
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
OFFICE OF INSPECTOR GENERAL Department of Homeland Security
Appendix C Report Distribution
Department of Homeland Security
Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison
Office of Management and Budget
Chief Homeland Security Branch DHS OIG Budget Examiner
Congress
Congressional Oversight and Appropriations Committees
12 wwwoigdhsgov OIG-15-91
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305
ADDITIONAL INFORMATION AND COPIES
To view this and any of our other reports please visit our website at wwwoigdhsgov
For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig
OIG HOTLINE
To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at
Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305