Data and Model Validation for Suspicious

Transaction Monitoring Systems3:20p| 27-September-2016

Bao Nguyen, Director of Regulatory Compliance, Kaufman, Rossin & Co.

Scott Nathan, EVP, BSA Officer, BankUnited, N.A.

Ivan A. Garces, Principal, Kaufman, Rossin & Co.

Daniel Haggerty, BSA/AML Examiner, Federal Reserve Bank of Atlanta

• Introductions

• Importance of system model validations and

best practices

• View from advisory firm perspective

• View from financial institution perspective

• View from regulator perspective

• Key takeaways and questions

Ivan A. Garces

Principal

Risk Advisory Services Practice Leader

Kaufman, Rossin & Co.

4

• Banks heavily rely on automated systems to

identify potential suspicious activity.

• Scenarios are based on judgmental and

quantitative considerations.

• Validation is an ongoing processes intended to

verify whether the suspicious activity monitoring

system is performing effectively and efficiently.

• The OCC and the Federal Reserve published

“Supervisory Guidance on Model Risk Management” (OCC 2011-12 & SR 11-7).

Validation, What Is It & Why Do We Need It?

5

• Evaluation of Conceptual Soundness

– Understand risk profile (customers, products

& services, geographies)

– Historical SAR experience

– GAP analysis

– Methods, assumptions and variables used

– System functionality & limitations, if any

Validation – Does Your System Make Sense?

6

• Data Input & Integrity

– Enterprise wide

– Customer

Information

– Transaction Data

– Data reconciliation

– Data mapping

Validation – Data Integrity

Ensure all appropriate customers & transactions are flowing into monitoring system

Core System Account/Transaction

Monitoring System

Wire System

Domestic and

International Wires

Profile

Exceptions

Cash Transaction

Aggregation

Wealth

Management System

CDD/EDD System

Expected Activity

Profiles

Ad-Hoc Reports

Flags/Alerts

7

• Evaluation of filtering criteria (rules, parameters

and thresholds)

– Should be commensurate with risk profile.

– Statistical analysis (customer & transaction

segmentation).

– Transaction testing to validate parameter

settings (ATL/BTL).

– Consider reasonableness of alert scoring.

– Evaluate efficacy of modifications to rules,

parameters & thresholds.

Validation – Testing the Parameters

8

Example – Rule captures any

business customer conducting

greater than $500,000 in wire

transfers (in/out) within a 30

day period.

82.7% of Avg.

Wire Activity

Above the Line /

Below the Line

Testing

Dollar

Threshold Percentile

Number of

Alerts

% of

Customers

Captured

% of Total

Wire Activity

Captured

less 5 percentile $ 357,444 80 154 20.0% 87.8%

less 2 Percentile $ 422,416 83 105 17.0% 84.9%

Current threshold $ 500,000 85 81 15.0% 82.7%

plus 2 percentile $ 645,579 87 46 13.0% 78.6%

plus 5 percentile $ 802,241 90 34 10.0% 74.3%

Business customers with Wire Transfers greater than $500,000

Validation – Rules Testing

9

• Evaluation of Reporting Process

– Back-testing of logic

– Productivity ratios

• Alert to RFI %

• Alert to investigation %

• Alert to SAR %

Validation – Does it Work?

10

Validation – Documentation

Model Inventory

Monitoring systems

Policies &

Procedures

Methodology &

approach

Types & source of

inputs

Data mapping

Data reconciliation

Model Inventory

Parameters & settings

Roles &

responsibilities

Reports

Date of last

assessment

Results of last

assessment

Limitations

Daniel HaggertySenior BSA/AML Examiner

Federal Reserve Bank of Atlanta

12

Validation for Suspicious Transaction

Monitoring

• FFIEC BSA/AML Exam Manual - Automated Account

Monitoring

• Parameters and filters should be reasonable and

tailored

• After parameters and filters have been developed,

they should be reviewed before implementation

• Once established, the bank should review and test

system capabilities and thresholds on a periodic

basis.

• The monitoring system’s programming methodology

and effectiveness should be independently validated

to ensure that the models are detecting potentially

suspicious activity.

13

FFIEC BSA/AML Examination Procedures:

Suspicious Activity Reporting

Step 5: Identify the types of customers,

products, and services that are included

within the surveillance monitoring system.

Considerations:

• Coverage of Products and Services

• Inventory of Bank’s Technology Systems

• Data Mapping

• Transaction Mapping

14

FFIEC BSA/AML Examination Procedures:

Suspicious Activity Reporting

Step 6: Identify the system’s methodology for

establishing and applying expected activity or profile

filtering criteria and for generating monitoring reports.

Determine whether the system’s filtering criteria are

reasonable.

Considerations:

• Money Laundering and Terrorist Financing Red

Flags (FFIEC BSA/AML Examinational Manual –

Appendix F)

• Support for Filtering Criteria

• Data Integrity of Key Fields

15

FFIEC BSA/AML Examination Procedures:

Suspicious Activity Reporting

Step 7: Determine whether the programming

of the methodology has been independently

validated.

Considerations:

• Roles and Responsibilities

• Frequency

• Scope

16

FFIEC BSA/AML Examination Procedures:

Suspicious Activity Reporting

Step 8: Determine that controls ensure

limited access to the monitoring system and

sufficient oversight of assumption changes.

Considerations:

• Change Management Framework

• Oversight

• Policies and Procedures

• Documentation

Scott Nathan

EVP, BSA Officer, Director of Financial

Crimes Risk Management

BankUnited, N.A.

18

Data Inputs

Customer

Transactions

Accounts

Data Sources

AML Rules Engine

Segmentation and

Thresholds

Txn Monitoring

Alert Risk Scoring &

Analytics

Alert Alert

Alert Triage

Alert

Processing

ETL Logic

Output & Investigations

Rule Responsive

Quality Assurance

MI &

Reporting

Alert Disposition

Alert Case

SAR

• Understand sources and logic used• Data integrity and quality checks• ETL methodology and testing

• Model translations• Typology mapping• Population stats and segmentation• Tuning and analytics

• Outcome assessments• Rule responsiveness• MIS reporting• Quality assurance

1

2

3

Three Primary Components of Successful

Validation

19

• Validation / tuning processes are inconsistent between banks; model behavior will differ .

• Disconnect between “independent assessment,” “tuning” and “model validation” exercises.

• Consulting firms are not able to form an official opinion regarding outcomes in a manner useful to management or internal audit.

• Lack of visibility around system performance across data sets.

• OCC has yet to publish exam procedures specific to model validation.

• Definition of rules/models when translated from code to ‘English’ often differ from management expectations.

• Regulators are the only ones with perspective into what works and what doesn’t across the footprint.

Challenges with ‘Validation’

20

Typical Model Validation Exercise

Task Description Challenge

1 Identify and source dataLack of data governance strategy or resources

with knowledge of environments

2 Assess models/scenarios Understanding model functionality

3 Develop code to replicate model output Understanding model structures

4 Reconciliation of model alert output Understanding data architecture

5 Model threshold evaluationEvaluation of risk assessment and underlying

population statistics

6 Quality assurance reviewCooperation between parties with open

communication

21

Model Assessment Lifecycle

• The following six functions are representative of the model risk management lifecycle within our BSA Analytics department

Product & Service Inventory

Typology Mapping

Coverage Gap Assessment

Data Source Reconciliation

Source to Target Mapping

ETL Process Evaluations

Model Prioritization Data Assessment

Functional Requirements

Logic Definitions and Testing

Output Evaluations

Model Review

Data Segmentation

Threshold Tuning

Sensitivity Assessments

Model Optimization

Householding

Scoring/Triage

Case Enrichment

Event Prioritization

Performance Evaluations

Productivity Ratios

Logic Enhancement

Perpetual Model Management

1 2 3

4 5 6

22

Source to Target Assessment

• Defining, evaluating and testing source system data and transaction code impact is critical and requires dedicated BSA technology resources.

• We manage and evaluate over 560 active tran codes in our core systems and their downstream impact on the monitoring platform.

• System model code sub mappings (‘buckets’)

• Direction of payments and need for custom ETL for mixed trxn groups

• File enrichment programs (wire, ACH, trade)

• Monitoring environment – perpetual control evaluation model (ETL, FTP jobs)

23

Sample SQL Logic Evaluation

WITH TRANS

AS

(

SELECT AC.account_key,

AC.acct_first_name,

AC.acct_last_name,

AC.acct_num,

AC.acct_type_cd,

AC.acct_open_date,

trn.acct_currency_amount,

trn.transaction_date_time,

trunc(trn.transaction_date_time,'Month') AS transaction_date,

CASE WHEN EXTRACT(MONTH FROM

trn.transaction_date_time) = 3 AND EXTRACT(YEAR FROM

trn.transaction_date_time) = 2015 THEN 1 ELSE 0 END AS

is_current_month

FROM AML_STAGING_DB_TEMP.ACCOUNT AC

INNER JOIN

AML_STAGING_DB_.................................................................

…………………………………………………………………….

OK

X

SQL Statement Logic Gap Assessment

Validation of known logic defect associated with historical behavior model

24

Key Optimization and Validation Practice

• Develop data model to support the tuning and optimization process inclusive of the ability to traverse in multiple dimensions between ‘alerts’ and SARs.

FIU Data Model Intelligent Suppression

25

Key Takeaways

• Examiners are increasing their focus on BSA/AML data validation

• Understanding the filters in your system and how it works is critical to assessing the effectiveness of your monitoring program

• Select a team who understands the system, underlying data models, and model validating process

• Follow the data – ensure the mappings and structures make sense

• Document rationale behind your existing rules and thresholds, and that filtering criteria and parameters are reflective of your institution’s risk.

26

Please rate this session now on the

ACAMS Conferences App

Session & Speaker Feedback

From App homepage:

• Click on session

• Find bubble or ”thumbs up” icon

• Rate session and speakers

THANK YOU!