Celso Mug Nela

7/28/2019 Celso Mug Nela

1/20

Validation in Risk Management:Approaches, Issues, and Observations

Kenneth G. Fulton

Division of Banking Supervision and Regulation

Board of Governors of the Federal Reserve System

**************************Febraban 2nd Annual Operational RiskConference

June 5, 2009


2/20

Agenda

z Validation more than models

z U.S. Final Rule

requirementsz Framework validation

z Data validation

z Model Validation

z Supervisory review

z Pillar 2

2

I think you should be more

explicit here in step number 2

This presentation contains the views and

opinions of the speaker, and does not

represent a policy statement of the FederalReserve Board of Governors.


3/20

Validation

z Validation in this context is a set of processesor activities designed to assess whether riskmanagement tools are working as intended

Applies to quantitative and qualitative tools (not justmodels)

z Validation is a control mechanism that helps

to assess uncertainty in risk management Tools for assessing risk always contain inherent

uncertainty

Validation helps minimize uncertainty (but cannotremove it)

3


4/20

Characteristics of Validation

Validation:

z Should provide an effective challenge to risk

management toolsz Does not rely on a single method

z Is a process, not an event

z Should cover all aspects of risk management

(including underlying components)

z Requires judgment and is a combination ofquantitative and qualitative assessment

4


5/20

Governance Framework

5


6/20

Components of Validation

z Validation activities generally fall into threecategories:

Review of the development process and conceptual foundation

Ongoing monitoring of process, which includes verifyingsoundness of approach and comparing to benchmarks

Analysis of outcomes (such as backtesting)

z

All three types of validation activities should beapplied to all elements of the risk measurement

process

Applied to the process that translates inputs into outputs, as wellas to the inputs and outputs themselves

z Key exercises for analyzing and checking outcomes

include sensitivity analysis and stress testing

6


7/20

Keys to Successful Validation

z Senior management is responsible for ensuring thatvalidation is working properly Through internal audit or other means

Should set clear expectations for acting on validation results

z Validation tends to be more effective if consideredbefore tools are developed not after the fact

z Good practice to have a firm-wide validation policy

z Many institutions have various levels of validation(conceptual / partial / full) Can be appropriate when full validation is not possible

But lack of full validation should always be transparent

If a tool has not been fully validated, there should be greateruncertainty about the estimates it produces

7


8/20

Keys to successful validation (cont.)

z Objectivity in validation is highly important Does not necessarily have to mean structural independence

But less structural independence may call for more controls toalleviate potential conflicts of interest

On its own, independent structure does not translate to effectivevalidation

z Documentation is vital to good validation Without it, evidence of objective validation is lacking

z Effective validation requires competent staff with theright skills, the right resources, the right incentives,and appropriate stature within the organization

Need ability to effectively challenge developers Need to identify issues and have a process for resolution

8


9/20

Model Validation Techniques

z Benchmarking

Benchmarking involves assessing the consistency ofthe estimated parameters with those obtained by other

estimation techniques, and potentially using other datasources.

z Backtesting

Backtesting generally involves comparing realized (expost) values with estimated (ex ante) parameters for acomparable and homogeneous data set.

z Judgmental Comparison

Comparison with other judgments and models

Comparison with other external information9


10/20

Validating Vendor Models

z Vendor models should be subject to the samevalidation standards as internal models

Validation includes not just the vendor model itself

9but also application of the vendor model to theindividual bank, its exposures, and risk profile

Documentation standards also apply Simple market acceptance does not suffice

Two key questions regarding vendor models:

9Does the model do what it is designed to do?9Does the model do what the bank is employing it to

do?

10


11/20

U.S. Final Rule Requirements

The [bank] must validate, on an ongoing basis, its advancedsystems. The [bank]s validation process must beindependent of the advanced systems development,

implementation, and operation, or the validation processmust be subjected to an independent review of itsadequacy and effectiveness. Validation must include:

i. An evaluation of the conceptual soundness of (includingdevelopmental evidence supporting) the advancedsystems;

ii. An ongoing monitoring process that includes verification ofprocesses and benchmarking; and

iii.An outcomes analysis process that includes back-testing.

11


12/20

Framework Validation

Examples:

z A broad assessment of the conceptual soundness of theAMA framework (e.g., Do the tools, processes, and

oversight systems the institution has designed andimplemented to measure and manage operational risk fitwith the quantitative techniques designed to quantify

operational risk?).z Are all the risk management tools subject to ongoing

monitoring and evaluation of their effectiveness? Does

the institution have robust mechanisms to periodically testwhether a particular tool or process remains longereffective?

12


13/20

Data Validation

Examples:

z Reconcile to source systems

z Review by a risk control functionz Review by internal audit

Does internal audits participation as an internal control

compromise its independence?

z Review by business line peers

z

Comparison with other data elementsz Comparison with experience or expertise

Backtesting of BEICFs to internal loss data

z Assumptions are also part of the data13


14/20

Framework for Supervisory Review

14


15/20

Pillar 2

15

Pillar 2:

Supervisory

Review and

Internal Capital

Adequacy

Assessment

Process

(ICAAP)

Pillar 3:

Market Discipline

Pillar 1:

Minimum Capital

Requirements


16/20

Pillar 2 Supervisory Review & ICAAP

Supervisory guidance issued 7-15-08

z Identify and measure material risk

Including, but not limited to: credit, market, operational,interest rate, and liquidity risks

z Set and assess capital adequacy goals in

relation to riskz Ensure the integrity of the ICAAP

ICAAP is not just an economic capital calculation

16


17/20

Implications for Operational Risk

z Overall process for operational risk may bebased heavily on AMA

Pillar 2 approach may use a different confidencelevel (99.9th percentile for Pillar 1) or insuranceoffset (limited to 20% of operational risk capital for

Pillar 1)z The AMA is not as rules-based as the

AIRB

No prescribed formulas

Institutions design framework that best fits theirorganization

17


18/20

Pillar 2 Validation Required

z Each bank should ensure that the componentsof its ICAAP, including any models and their

inputs, are subject to the banks validation

policies and procedures.

z

A bank should be cognizant that bothquantitative and qualitative approaches have

their own inherent biases and assumptions

that affect risk assessment. Accordingly, abank should recognize the biases and

assumptions embedded in, and the limitations

of, the approaches used. 18


19/20

Supervisory References

z Risk Model Validation ; OCC Bulletin 2000-16http://www.occ.treas.gov/OCC_current.htm

z Model Governance ; FDIC Supervisory Insights, Winter 2005

http://www.fdic.gov/regulations/examinations/supervisory/insight

s/siwin05/index.html

z Guidelines on the implementation, validation and assessment of

Advanced Measurement (AMA) and Internal Ratings Based (IRB)

Approaches, Committee of European Bank Supervisors, April2006 http://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-

d652fa6ed1ec/GL10.aspx

z Supervisory Guidance: Supervisory Review Process of Capital

Adequacy (Pillar 2) Related to the Implementation of the Basel IIAdvanced Capital Framework, July 2008

http://www.federalreserve.gov/newsevents/press/bcreg/bcreg200

80715a1.pdf

19
http://www.occ.treas.gov/OCC_current.htmhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.occ.treas.gov/OCC_current.htm


20/20

Kenneth Fulton

[email protected]

Celso Mug Nela

Documents

Transcript of Celso Mug Nela