OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal...

15
CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls May 11, 2015 OIG-15-91

Transcript of OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal...

Page 1: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

CBP is on Track to Meet ACE Milestones but It Needs to Enhance Internal Controls

May 11 2015 OIG-15-91

DHS OIG HIGHLIGHTS CBP is on Track to Meet ACE Milestones but It Needs

to Enhance Internal Controls

May 11 2015

Why We Did This The Automated Commercial Environment (ACE) is the commercial trade system designed to automate border processing to enhance border security and foster our Nationrsquos economic security ACE is part of a multi-year US Customs and Border Protection (CBP) modernization effort that is being deployed in phases and must be completed by December 2016 We conducted the audit to determine whether CBP is on track to meet its milestones for the implementation of ACE

What We Recommend We made one recommendation to continuously assess evaluate and update internal controls to include a risk assessment that identifies potential data reliability gaps and develop and implement specific measurable achievable relevant and time-sensitive performance measures This recommendation when implemented should improve the efficiency and effectiveness of the program

For Further Information

Contact our Office of Public Affairs at (202) 254-4100 or email us at DHS-OIGOfficePublicAffairsoigdhsgov

What We Found CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured to a rapid deployment strategy called Agile Currently in large part due to the implementation of Agile CBP is on track to meet its milestones for the deployment of ACE However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program Similar internal control issues were identified in a 2014 KPMG financial statement audit If these weaknesses continue and internal controls do not keep pace with Agile and the rapid implementations of the ACE program deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBP Response In its response to our draft report CBP reported that it appreciated the Office of Inspector Generalrsquos (OIG) recognition that the ACE program is on track to meet its implementation milestones and OIGrsquos positive conclusion that the Agile development methodology improves the effectiveness and efficiency of the ACE development process However CBP did not concur with our report recommendation

wwwoigdhsgov OIG-15-91

vttllf r

~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security

Washington DC 20528 wwwoigdhsgov

MAY 11 2015

MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection

FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits

SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls

For your action is our final report CBP is on Track to Meet ACE Milestones but

It Needs to Enhance Internal Controls We incorporated the formal comments

provided by your office

The report contains one recommendation aimed at improving the Automated

Commercial Environment program Your office did not concur with the

recommendation Based on information provided in your response to the draft

report we consider the recommendation open and unresolved As prescribed

by the Department of Homeland Security Directive 077-01 Follow-Up and

Resolutions for the Office of Inspector General Report Recommendations within

90 days of the date of this memorandum please provide our office with a

written response that includes your (1) agreement or disagreement

(2) corrective action plan and (3) target completion date for the

recommendation Also please include responsible parties and any other

supporting documentation necessary to inform us about the current status of

the recommendation Until your response is received and evaluated the

recommendation will be considered open and unresolved

Consistent with our responsibility under the Inspector General Act we will

provide copies of our report to congressional committees with oversight and

appropriation responsibility over the Department of Homeland Security We will

post the report on our website for public dissemination

Please call me with any questions or your staff may contact Paul Wood Acting

Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also

send your response to OIGAuditsFollowup(c~oi~ dhsbullgov

Attachment

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo

Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage

The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status

In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE

wwwoigdhsgov 2 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million

Results of Audit

CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues

If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBPrsquos ACE Program on Track

CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams

CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which

1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion

wwwoigdhsgov 3 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 2: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

DHS OIG HIGHLIGHTS CBP is on Track to Meet ACE Milestones but It Needs

to Enhance Internal Controls

May 11 2015

Why We Did This The Automated Commercial Environment (ACE) is the commercial trade system designed to automate border processing to enhance border security and foster our Nationrsquos economic security ACE is part of a multi-year US Customs and Border Protection (CBP) modernization effort that is being deployed in phases and must be completed by December 2016 We conducted the audit to determine whether CBP is on track to meet its milestones for the implementation of ACE

What We Recommend We made one recommendation to continuously assess evaluate and update internal controls to include a risk assessment that identifies potential data reliability gaps and develop and implement specific measurable achievable relevant and time-sensitive performance measures This recommendation when implemented should improve the efficiency and effectiveness of the program

For Further Information

Contact our Office of Public Affairs at (202) 254-4100 or email us at DHS-OIGOfficePublicAffairsoigdhsgov

What We Found CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured to a rapid deployment strategy called Agile Currently in large part due to the implementation of Agile CBP is on track to meet its milestones for the deployment of ACE However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program Similar internal control issues were identified in a 2014 KPMG financial statement audit If these weaknesses continue and internal controls do not keep pace with Agile and the rapid implementations of the ACE program deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBP Response In its response to our draft report CBP reported that it appreciated the Office of Inspector Generalrsquos (OIG) recognition that the ACE program is on track to meet its implementation milestones and OIGrsquos positive conclusion that the Agile development methodology improves the effectiveness and efficiency of the ACE development process However CBP did not concur with our report recommendation

wwwoigdhsgov OIG-15-91

vttllf r

~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security

Washington DC 20528 wwwoigdhsgov

MAY 11 2015

MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection

FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits

SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls

For your action is our final report CBP is on Track to Meet ACE Milestones but

It Needs to Enhance Internal Controls We incorporated the formal comments

provided by your office

The report contains one recommendation aimed at improving the Automated

Commercial Environment program Your office did not concur with the

recommendation Based on information provided in your response to the draft

report we consider the recommendation open and unresolved As prescribed

by the Department of Homeland Security Directive 077-01 Follow-Up and

Resolutions for the Office of Inspector General Report Recommendations within

90 days of the date of this memorandum please provide our office with a

written response that includes your (1) agreement or disagreement

(2) corrective action plan and (3) target completion date for the

recommendation Also please include responsible parties and any other

supporting documentation necessary to inform us about the current status of

the recommendation Until your response is received and evaluated the

recommendation will be considered open and unresolved

Consistent with our responsibility under the Inspector General Act we will

provide copies of our report to congressional committees with oversight and

appropriation responsibility over the Department of Homeland Security We will

post the report on our website for public dissemination

Please call me with any questions or your staff may contact Paul Wood Acting

Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also

send your response to OIGAuditsFollowup(c~oi~ dhsbullgov

Attachment

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo

Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage

The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status

In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE

wwwoigdhsgov 2 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million

Results of Audit

CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues

If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBPrsquos ACE Program on Track

CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams

CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which

1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion

wwwoigdhsgov 3 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 3: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

vttllf r

~ ~~ OFFICE OF INSPECTOR GENERAL`yf~i T Department of Homeland Security

Washington DC 20528 wwwoigdhsgov

MAY 11 2015

MEMORANDUM FOR Brenda SmithAssistant CommissionerOffice of International TradeUS Customs and Border Protection

FROM Mark Bell ~ ~ ~~~-~Assistant Inspector General for Audits

SUBJECT CBP is on Track to Meet ACE Milestones but It Needs toEnhance Internal Controls

For your action is our final report CBP is on Track to Meet ACE Milestones but

It Needs to Enhance Internal Controls We incorporated the formal comments

provided by your office

The report contains one recommendation aimed at improving the Automated

Commercial Environment program Your office did not concur with the

recommendation Based on information provided in your response to the draft

report we consider the recommendation open and unresolved As prescribed

by the Department of Homeland Security Directive 077-01 Follow-Up and

Resolutions for the Office of Inspector General Report Recommendations within

90 days of the date of this memorandum please provide our office with a

written response that includes your (1) agreement or disagreement

(2) corrective action plan and (3) target completion date for the

recommendation Also please include responsible parties and any other

supporting documentation necessary to inform us about the current status of

the recommendation Until your response is received and evaluated the

recommendation will be considered open and unresolved

Consistent with our responsibility under the Inspector General Act we will

provide copies of our report to congressional committees with oversight and

appropriation responsibility over the Department of Homeland Security We will

post the report on our website for public dissemination

Please call me with any questions or your staff may contact Paul Wood Acting

Deputy Assistant Inspector General for Audits at (202) 254-4100 You can also

send your response to OIGAuditsFollowup(c~oi~ dhsbullgov

Attachment

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo

Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage

The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status

In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE

wwwoigdhsgov 2 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million

Results of Audit

CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues

If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBPrsquos ACE Program on Track

CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams

CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which

1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion

wwwoigdhsgov 3 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 4: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Background In support of the 1993 Customs Modernization Act US Customs and Border Protection (CBP) has been modernizing the business processes essential to securing US borders speeding the flow of legitimate shipments and targeting illicit goods that require scrutiny The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo

Executive Order 13659 mandates CBP to provide participating agencies the capabilities agreements and other requirements necessary to use the ITDS and supporting systems such as the Automated Commercial Environment (ACE) by December 31 2016 ACE is a key technology driver of these initiatives and will be the primary means of receiving usersrsquo standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export Once fully implemented ACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storage

The development of the ACE program began in 2001 In 2006 Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program These procedures required CBP to work with the Department of Homeland Securityrsquos Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review In 2009 CBP presented a release to the PADR team for evaluation and approval This review was expanded to encompass aspects of the overall ACE program The PADR team identified deficiencies such as significant delays cost overruns and unclear system requirements According to CBP the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development The PADR team recommended that ACE not continue beyond production readiness review The Department of Homeland Securityrsquos (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project which placed the program into breach status

In September 2012 the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012 CBP started a pilot program in 2013 which restructured its process to Agile a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability In June 2013 the program was removed from breach status and ACE

wwwoigdhsgov 2 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million

Results of Audit

CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues

If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBPrsquos ACE Program on Track

CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams

CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which

1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion

wwwoigdhsgov 3 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 5: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

development restarted using Agile All costs prior to the completion of the pilot program were considered sunk1 costs CBP spent approximately $32 billion on the development of the ACE program from fiscal year 2001 through July 2013 when it was restructured CBPrsquos fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million

Results of Audit

CBP is on track to meet its milestones for the implementation of the ACE program Currently CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time mainly due to the recently implemented rapid deployment strategymdashAgile However CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program Specifically CBP has not conducted risk assessments to identify potential gaps in data reliability and has not fully developed and implemented performance measures for the program In 2014 KPMG conducted a financial statement audit and identified similar internal control issues

If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers development and deployment schedules could be adversely impacted resulting in missed future deadlines and compromised effectiveness of ACE

CBPrsquos ACE Program on Track

CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31 2016 Presidential mandate In 2013 CBP restructured its development process for the ACE program and changed to Agile This rapid deployment strategy supports the practice of shorter software delivery more oversight and accountability and allows more flexibility to accommodate changing requirements and shifting priorities Specifically the strategy calls for delivery of software in small short increments and emphasizes collaborative teams

CBP uses a 13-week cycle or increment that is broken down into 1 week of planning and six 2-week-long development periods called ldquosprintsrdquo CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints which

1 CBP defines sunk costs as all funds obligated prior to July 2013 which includes $464 million in Agile development costs This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status When the future costs (through FY 2026) and sunk costs are combined the estimated life cycle cost is approximately $423 billion

wwwoigdhsgov 3 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 6: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

combine into a larger body of work at the end of each increment This process provided CBP with flexibility and oversight of development For example CBP required development teams to update management daily and demonstrate work completed biweekly The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances

According to CBPrsquos Increment and Deployment Schedule there are seven deployments (A through G) of ACE capabilities Deployments A B C and D were implemented on schedule with Deployment D being implemented in January 2015 CBPrsquos goal is to have all seven deployments completed by July 2016 which is 5 months before the mandated deadline CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline)

Risk Assessment Needed

CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them The internal control environment provides the structure to help an entity achieve its objectives According to the Committee of Sponsoring Organizationsrsquo Internal Control Integrated Framework internal controls keep an organization on course toward meeting goals and achieving its mission In addition GAOrsquos Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is

1 well designed and operated 2 appropriately updated to meet changing conditions and 3 providing reasonable assurance that the objectives of the agency are

being achieved

As a component of the internal control environment a risk assessment provides the basis for developing appropriate control activities Even though CBP did complete a Risk Management Plan it did not provide proof it was used to formally assess risk In addition the planrsquos focus identified risks in the development process and may not identify other risks to the system

During the Department of Homeland Security FY 2014 Integrated Audit KPMG tested key information technology controls for ACE KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments Additionally during our audit we found that CBP granted software developers access to the live database in order to update the system This may create a risk that developers have access to ACE without

wwwoigdhsgov 4 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 7: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

proper internal controls Without a risk assessment CBP may be unaware of potential risks or other data reliability concerns in ACE

Performance Measure Updates Needed

ACE performance measures under the rapid deployment strategy have not been fully developed and implemented CBP created measures for the original development strategy however it has not updated them to reflect the desired outcomes of the rapid deployment strategy For example CBPrsquos FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP Of the 10 targets 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard CBP began the rapid deployment strategy in 2013 therefore the performance measures were created for the original deployment strategy Specifically on the 2014 scorecard a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006 This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such may not be a relevant measure of the ACE program

CBPrsquos performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy The measures are not well defined or explained do not have completion criteria timeframes or the level of specificity needed to show how they prove an accomplishment Performance measures are an important tool to ensure a project is meeting its stated objectives and should be specific measurable achievable relevant and time-sensitive (SMART) However CBP has recently drafted new performance measures which were not ready for review

Recommendations

Recommendation We recommend that the Assistant Commissioner Office of International Trade US Customs and Border Protection continuously assess evaluate and update internal controls during each 13-week development increment Specifically

a Conduct a risk assessment to identify potential data reliability gaps and b Develop and implement specific measurable achievable relevant and

time-sensitive (SMART) performance measures

wwwoigdhsgov 5 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 8: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Management Comments and OIG Analysis

In its response to our draft report CBP did not concur with our report recommendation A summary of CBPrsquos response and our analysis follows We have included a copy of the management comments in their entirety in appendix A CBP also provided technical comments to our draft report

Management Comments

CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program According to CBP the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features In addition risk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process Finally according to CBP program officials have provided OIG with many documents to demonstrate that well-developed SMART performance measures are being used along with the implementation of other internal controls

OIG Analysis

In its response to our draft report CBP stated ldquoRisk assessments are being completed before each deployment as part of the CBPrsquos Production Readiness Review (PRR) process and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR processrdquo However as stated this review is conducted prior to deployment

CBP further stated in its technical comments that ldquoAfter deployment we complete risk assessments of data reliability through validations and editsrdquo OIG contends that doing ldquovalidation and editsrdquo is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses Per the Government Accountability Officersquos (GAO) Standards for Internal Control in the Federal Government management should identify analyze and respond to risks related to achieving the defined objective This would include assessing inherent and residual risk internal and external factors potential of fraud magnitude of impact likelihood of occurrence and nature of risk among other factors

wwwoigdhsgov 6 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 9: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

In fact according to CBPrsquos own technical comments ldquoThe ACE Program did not identify data reliability as a risk and [as] such data reliability is not part of the ACE Program risk reportrdquo As detailed in the Background section of this report ldquoACE will become the central trade data collection system for all Federal agencies and the single point of access for this data which includes data collection processing dissemination and storagerdquo As the ACE program will be used to collect duties taxes and fees and to make other key decisions the OIG contends that data reliability is the cornerstone of the ACE program Furthermore given the multitude of data reliability issues within DHS found during prior OIG and GAO audits we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps

In addition the OIG still recommends that CBP develop and implement SMART performance measures In its response to our draft report CBP stated ldquoCBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilizedhelliprdquo From a review of CBPrsquos provided documents we determined that those documents are merely data collection spreadsheets and not performance measures The measures are not specific or time sensitive In other words it is unclear what the goal is and when it is to be achieved

In developing an effective risk assessment GAO states that management should ldquoDefine objectives in specific terms so they are understood at all levels of the entity This involves clearly defining what is to be achieved who is to achieve it how it will be achieved and the time frames for achievementrdquo CBPrsquos Performance Measurement Indictor spreadsheet does not have SMART performance measures

Therefore the OIG considers this recommendation unresolved and open We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures We will close the recommendation when we receive appropriate completion documentation

Objective Scope and Methodology

The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978 This is one of a series of audit inspection and special reports prepared as part of our oversight responsibilities to promote economy efficiency and effectiveness within the Department

We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation To achieve our audit objective we identified and reviewed applicable Federal laws regulations

wwwoigdhsgov 7 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 10: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

and CBP policies and procedures regarding the ACE program The audit covered the development of the ACE program from January 2013 through September 2014

We interviewed CBP personnel responsible for the development management and administration of ACE including key stakeholders from the ACE Business Office Office of Information amp Technology Office of International Trade Office of Technology Innovation amp Acquisition Office of Field Operations Outcomes and Analysis Branch and Trade Management and Information Division We also interviewed subject matter experts and contractors responsible for ACE development Additionally we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies the Environmental Protection Agency and Food Safety and Inspection Service

We performed limited survey work to ensure the accuracy and completeness of information used in this audit Additionally we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives

We conducted this performance audit between June and October 2014 pursuant to the Inspector General Act of 1978 as amended and according to generally accepted government auditing standards Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives We believe that the evidence obtained provides a reasonable basis for our findings and conclusions

Office of Audits major contributors to this report are Brooke Bebow Director Patrick Tobo Audit Manager Priscilla Cast Co-Auditor-in-Charge Anthony Colache Co-Auditor-in-Charge Tia Jackson Program Analyst Frank Lucas Auditor Sandra Ward-Greer Auditor Kevin Dolloson Communications Analyst and Marissa Weinshel Independent Referencer

wwwoigdhsgov 8 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 11: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix A Management Comments to the Draft Report

wwwoigdhsgov 9 OIG-15-91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 12: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

wwwoigdhsgov 10 OIG-15-91

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 13: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFF

ICE

OF

INSP

EC

TOR

GEN

ER

AL

Dep

artm

ent

of H

omel

and

Sec

uri

ty

App

endi

x B

A

CE

Pro

gram

Tim

elin

e

Oct

ober

201

0 O

ctob

er 2

011

Apr

il 20

01

June

201

3 Ja

nuar

y 20

15

Janu

ary

2016

D

ecem

ber

2016

CB

P in

itia

tes

DH

S A

cqu

isit

ion

O

rigi

nal

AC

Ere

visi

on o

f AC

E

Rev

iew

Boa

rd

CB

P ob

tain

s D

eplo

ymen

t D

Dep

loym

ent F

Man

date

d D

evel

opm

ent

plan

nin

gde

cisi

on to

ap

prov

al fo

r

expe

cted

expe

cted

to

bede

adlin

e fo

r A

CE

be

gin

s do

cum

enta

tion

pr

ocee

d A

gile

to

be

com

plet

ed

com

plet

ed

com

plet

ion

June

200

9 D

ecem

ber

2010

Ja

nuar

y 20

13

May

201

4 Ju

ly 2

015

July

201

6

Prog

ram

C

BP

star

ts

Dep

loym

ents

C

BP

appr

oves

Dep

loym

ent E

Dep

loym

ent

G

Ass

essm

ent

Agi

le p

ilot f

orre

vise

d ac

quis

itio

n

AB

an

d C

had

ex

pect

ed t

o be

expe

cted

to

beD

esig

n R

evie

wA

CE

st

rate

gy

been

rel

ease

d by

com

plet

ed

com

plet

ed

Team

th

is d

ate

reco

mm

ends

A

CE

not

co

nti

nu

e

Sour

ce C

reat

ed b

y D

HS

OIG

11

ww

wo

igd

hsg

ov

OIG

-15-

91

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 14: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

OFFICE OF INSPECTOR GENERAL Department of Homeland Security

Appendix C Report Distribution

Department of Homeland Security

Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director GAOOIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison

Office of Management and Budget

Chief Homeland Security Branch DHS OIG Budget Examiner

Congress

Congressional Oversight and Appropriations Committees

12 wwwoigdhsgov OIG-15-91

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305

Page 15: OIG-15-91 - CBP is on Track to Meet ACE Milestones, but It ...€¦ · January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before

ADDITIONAL INFORMATION AND COPIES

To view this and any of our other reports please visit our website at wwwoigdhsgov

For further information or questions please contact Office of Inspector General Public Affairs at DHS-OIGOfficePublicAffairsoigdhsgov Follow us on Twitter at dhsoig

OIG HOTLINE

To report fraud waste or abuse visit our website at wwwoigdhsgov and click on the red Hotline tab If you cannot access our website call our hotline at (800) 323-8603 fax our hotline at (202) 254-4297 or write to us at

Department of Homeland Security Office of Inspector General Mail Stop 0305 Attention Hotline 245 Murray Drive SW Washington DC 20528-0305