www.fico.com Make every decision countTM
INSIGHTS WHITE PAPER
Number 82
Reducing Regulatory Drag on Analytics Teams Automated workflows standardize and speed model management processes
Banking regulators are increasing scrutiny of analytic models, peeling back layers of the onion with
probing questions. They want to know not only how models affect credit policies and customer
decisions, but about the processes used for developing, validating, deploying and updating them.
Banking executives, increasingly aware of the full dimensions of model risk, are also asking pointed
questions.
Finding answers can add drag to the performance of analytics teams—even pulling them away from
high-value work that leads to competitive advantage.
To improve compliance and response time to detailed questions, leading banks are implementing
formal model management processes throughout the analytic lifecycle. But while best practices may
be understood, they can be challenging to deploy consistently
across analytic teams. It’s also difficult to know if they’re being
followed at the right level of granularity, such that no matter
where regulators probe—and even with analytic staff turnover—
all questions can be readily answered.
This white paper examines how automated, configurable model workflow tools promote process
consistency and accountability. We show how banks are using workflow at enterprise and
departmental levels to improve model governance without creating extra work for analytic teams.
In fact, by orchestrating model processes while automatically capturing key artifacts, decisions and
sign-offs, workflow can lead to better model performance. Analytic teams are freed to spend more
of their time creating and updating models.
We’ll cover:
• Turning work friction into work flow
• Creating standard processes that fit the needs of diverse teams
• Tracking model and characteristic lineages
Improve compliance while spending 75% less time on it.
Reducing Regulatory Drag on Analytics Teams
INSIGHTS WHITE PAPER
November 2014 www.fico.com page 2
Banks, long at the vanguard of data analytics for business, have continued to expand their use
of models. Predictive scorecards and other models for anticipating and responding to customer
behavior now play a central role in every area of operational decision making. The sheer quantity
of these models is increasing rapidly.
What hasn’t advanced as quickly is model management. An August 2014 article by
Butler Analytics pointed out that “some banks simply do not know how many models are
actually deployed.” In other cases, model information is in so many places that preparing for
audits or answering regulator inquiries becomes extremely labor-intensive. The Butler article
reports that “the modeling staff in one major US bank now spend 80% of their time meeting
regulatory requirements, detracting from much needed new model development.”
While this is an extreme case, every bank is seeing an impact on the productivity and
performance of its analytics teams. Both those building models and those compiling reports
and answering inquiries have more work to do. And where one group is charged with both
duties, compliance tasks can siphon away analytic resources from work that could be producing
significant value for the bank.
An executive in the mortgage division of one bank told FICO that, for a period of time, 25% of its
analytics workforce had to be diverted to collecting, preparing and reporting on data required by
regulators—costing the bank tens of millions of dollars.
Yet some banks are moving ahead of the curve. They’re improving their ability to answer
questions about analytics while lightening the burden on their analytics teams. In fact, a bank
devoting 80% of modeler time to regulatory requirements could reduce that expenditure to 20%
or less.
Model Management—Getting Ahead of the Curve
Over the past five years, analytic excellence has become a core requirement in today’s financial market… Modeling touches virtually every decision of the bank… IDC Financial Insights A Framework for Model Governance, June 2013
“Organizations today face heavy regulatory pressures…To meet these challenges and mitigate risk, they need model management solutions that can reduce resources required to complete compliance
audits, and encompass the full model lifecycle and risk-management continuum...” ” Peyman Mestchian Managing Partner at Chartis
www.fico.com page 3
Reducing Regulatory Drag on Analytics Teams
INSIGHTS WHITE PAPER
This more efficient, streamlined approach combines centralized model management with
automated, configurable workflow tools. As depicted in Figure 1, workflows enforce approved
processes at every stage of the model lifecycle. They capture granular information about what
decisions and actions are taken and why.
Populated by these workflows, the central repository maintains an inventory of all models in
operations and under development—their purpose, data types/sources, key assumptions,
exclusions, predictive characteristics,
segmentation schemes, where model is
being used, restrictions on usage, etc. The
model management solution also automates
scheduled validations and flags models for
review when stability or performance metrics
decline. This makes it much easier for banks to
maintain models at peak performance, as well
as spot and resolve any compliance issues.
Analytics teams accessing this shared resource
have detailed information at hand to explain,
for example, how they analyzed population
segments for a particular model—or across
multiple models—and defend their choice of
segmentation. They can readily justify why a
new predictive characteristic was added during
a model refresh and how it was developed.
They can provide regulators with evidence that
a rigorous analysis of all potential economic
drivers was performed for Long Run and
Downturn PD estimation.
Turning work friction into work flow
We’ve discussed how increasing regulatory
scrutiny has affected the work processes
of bank analytic teams. Now here’s the real
rub: This burden is settling on analytics
teams at exactly the moment when the
importance of what they do for helping banks
understand customers and create competitive
differentiation has become strikingly clear.
Banking executives want their analytic teams
to bring more powerful, complex analytics to
market as quickly as possible. But they also
need them to capture all of the details and
reasons along the way so that models and
their usage are transparent and explainable to
regulators, customers and executives.
FIGURE 1: BEING PREPARED TO ANSWER ANY AND ALL REGULATOR QUESTIONS
Banks can answer such questions with confidence when they have standardized, approved processes in place across the model lifecycle...
...and one place to access all process history details
Regulators are asking more detailed “peel back the onion” questions about models, such as:
When was this model redeveloped?
Why was this new characteristic added?
How was this characteristic calculated?
New customer characteristic:Ratio velocity retail purchases to velocity ATM withdrawals (Predictive of rising credit risk)
CEN
TRA
LIZE
DM
OD
EL M
AN
AGEM
ENT
Behavior Score Redevelopment Process History
VALIDATION
SUMMARY
PlansResultsMetadataVariables & RolesConfigurationProcessesModel Usage
DEFINITION
Models Alert Rulesets Validation Jobs Targets
WorkspaceDashboard AdministrationProcess AuthoringAnalytics Team A
Task Name AssigneeVariable Value
Task Activity
Independent review D. Berman IR_Approval_Status Approved
Develop model J. Kalabar Technical_Review Behavior Score Redevelopment Technical Review.doc
Data quality review R. Santos DQ_Status Approved
Assess data quality J. Kalabar DQ_Report Behavior Score Redevelopment Data Quality Report.docRole assignment L. Moreau Project_Team_Status Assigned
DQ_Comments Key fields look good. No problem that PromoCode is scarcely populated.
Model_Specs Behavior Score Redevelopment Specification.doc
IR_Comments Great work. Let’s proceed with implementation testing.
1
2
3
Model development documentation provides information on how the characteristic was developed as well as its predictive power.
UPDATED BEHAVIOR SCORE
Development Validations Redevelopment
BEHAVIOR SCORE LIFECYCLE
www.fico.com page 4
Reducing Regulatory Drag on Analytics Teams
INSIGHTS WHITE PAPER
Executives also want to get answers that allay their concerns about model risk. It’s not just
potential financial and reputational damage from regulatory noncompliance that’s worrisome,
but the opportunity costs of delayed model deployments and competitive impacts from
making decisions with underperforming models.
Automated workflow reduces the friction of these additional requirements for the entire
analytics team.
For model development: Workflow guides users through standard processes for
documenting actions and decisions at each step. (See Figure 2.) It eliminates manual
data input by automatically capturing segmentation criteria, test results and other key
information. It also orchestrates review cycles across departments, notifies stakeholders
and ensures approvals are obtained for moving on to the next step. Dashboards provide
at-a-glance status on the real-time progress of all models.
For regulatory and internal reporting and inquiries: Workflow feeds data about
model lifecycle processes into a centralized repository. Users compiling reports, preparing
for audits or responding to queries work faster and more efficiently with this single source
of all relevant information on models and modeling processes. They can very quickly, for
instance, examine a model’s most recent validation test. But they can also just as easily
peruse an entire process history encompassing every validation ever performed on that
model, and the results and consequent actions in each instance.
FIGURE 2: AUTOMATED WORKFLOW ORCHESTRATES MODEL LIFECYCLE PROCESSES WHILE CAPTURING DATA AND DOCUMENTATION FOR CENTRALIZED MANAGEMENT
Models Processes My TasksWorkspaceDashboard AdministrationProcess Authoring
Analytics Team A
Redevelop Behavior Model
Task: Data Quality Review
Your Actions
Using the form below, review the model data quality assessment and supporting documentation, and render an approval decision
Decision Action: Comments:
Supporting InformationModel Name:
Change Status:
Change Summary:
Model Owner:
Project Docments:
Comment History:
Behavior Score
Pending ReviewUpdating the model in response to validation
review recommendations and to incorporate
new transactional predictive characteristics
J. Kalabar
Behavior Score Redevelopment Techinical Review.docPlease note in the documentation which records were
used for the out-of-time validation of the new parameters
Approval Required:
Model review triggered by out-of-range results during periodic automated validation; recommenda-tion to redevelop
Process initiated by model manager
Email to analytics team leader: Tasks required: Project approval & role assignment
Email to model developer: Task required: Data quality assessment
Email to technical reviewer: Task required: Data quality approval
Data & documentationcaptured in shared repository
Development Validations Redevelopment
BEHAVIOR SCORE LIFECYCLE PROCESS MANAGEMENT
1
2 3 4
5
www.fico.com page 5
Reducing Regulatory Drag on Analytics Teams
INSIGHTS WHITE PAPER
Overall, this approach provides banks (and regulators) with
complete visibility into how each model is developed, deployed
and maintained. It ensures that the evidence banks need to
explain and defend their analytic choices is always fully captured
and readily accessible.
Creating standard processes that fit the needs of diverse teams
Banks vary greatly in how they use analytics across their
enterprise and the scope of their efforts to standardize lifecycle
model management processes. Here are some examples:
• One FICO client, a top-five US bank, is moving to centralize
management of every model across its vast enterprise.
• A leading Asian bank is initially focusing on ensuring its models
in development achieve Advanced Internal Rating Based status
under the Basel III global standard.
• A top-five Australian bank seeks to bridge current process
inconsistencies around model tracking and validation of Basel
rating models and decision models across different countries.
For any initiative, flexibility to align workflows with the needs
of analytic teams is essential. Managing best practices depends
partly on the types of analytics that teams are developing. For
instance, predictive models for forecasting customer behavior
have their own specific requirements and methodological pitfalls
to be avoided. So do descriptive models for improving population
segmentation and prescriptive models for recommending best
next actions. In addition, expert (judgment-based) models need
to be documented in very different ways than empirically derived
models. Requirements and methods also vary, of course, across
geographies and markets.
To accommodate this diversity, banks need workflow tools that
incorporate automated business rules management technology.
By authoring and changing business rules, analytics groups can
easily adjust workflows—within standardized parameters and
constraints—to their local needs.
At the same time, centralized repositories enable analytic
teams to share characteristic libraries and learn from each
other’s documentation and validation results. They also support
collaboration, where appropriate, across teams.
FICO® Model Central™ provides consistent end-to-end model
governance and a central repository for all of an organization’s analytic
models. The solution manages models created in any vendor’s tool,
showing their real-time status and monitoring their performance. New
advanced workflow capabilities in version 5.0 orchestrate and manage
model lifecycle processes.
Automated, configurable workflows notify process participants when
they have a new task and provide instant access to the information
needed to complete it. As tasks are performed, workflows capture
key steps, artifacts, decisions and sign-offs. Over time, Model Central
compiles a comprehensive audit trail for model lifecycle processes—
project initiation through data quality testing, model development,
implementation, deployment, validation, and updates or replacement.
Together, these advanced capabilities enable organizations to achieve
peak model performance, while providing the transparency and
accountability required by banking regulators and executives. Without
diverting analytics teams from more valuable work, they help reduce
model risk and raise compliance.
CENTRALIZING MODEL MANAGEMENT
ModelData Mart
Tracking
Monitoring
OngoingValidation
ManagementReporting
Alerts
DecisionSimulation
DecisionExecution
ScoringServices
DecisionOptimization
Development&
Calibration
Deployment&
Verification
ModelData Mart
ADVANCED
P
RO
FESSIONAL
DEC
IS
IONING
DEV
ELOPMENT
FO
UN
DA
TION
Reducing Regulatory Drag on Analytics Teams
INSIGHTS WHITE PAPER
For more information North America Latin America & Caribbean Europe, Middle East & Africa Asia Pacificwww.fico.com +1 888 342 6336 +55 11 5189 8222 +44 (0) 207 940 8718 +65 6422 7700 [email protected] [email protected] [email protected] [email protected]
FICO, Model Central and “Make every decision count” are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2014 Fair Isaac Corporation. All rights reserved.
4072WP 11/14 PDF
The Insights white paper series provides briefings on research findings and product development directions from FICO. To subscribe, go to www.fico.com/insights.
Tracking model and characteristic lineages
The concept of model lifecycle management is broader than the life of any particular model. As
analytics proliferate across organizations and the pace of change in financial services markets
accelerates, banks need to start thinking in terms of lineages.
Regulators may, for instance, ask pointed questions about why a retired model was replaced
with the current one, and which customer characteristics were given greater predictive weight
in the process. When a customer characteristic is changed by its author/owner, banks need to
know which models incorporate that characteristic so they can manage all downstream effects.
State-of-the-art model lifecycle management takes this broader view. Automated workflows
help banks capture a complete lifecycle history of all models and their components. For each
model, users can quickly track the lineage of any predictive customer characteristic—generated
during development, harvested from a previous model, taken from a shared library, etc. For each
characteristic, they can see everywhere it is currently used or was previously used—predictive
models, segmentation strategies, decision strategies, etc.
Another advantage of this approach is that banks have the opportunity to evaluate the value of
individual customer characteristics over time.
Increasingly far-reaching and detailed regulatory scrutiny is making it more important than ever
for banks to put standard, approved model management processes in place. At the same time,
banking executives want more visibility into and control over the full dimensions of model risk,
including both compliance exposure and performance issues.
Automated workflows that feed model lifecycle management solutions help banks lower model
risk by improving process consistency and accountability. And they do it without clipping the
wings of analytic teams—in fact, they offer efficiencies that can help them soar.
To learn more about best practices for model management, visit the FICO Blog and read these
Insights papers:
• Customer Centricity: Four Bank Success Stories (No. 78)
• Satisfying Customers and Regulators: Five Imperatives (No. 75)
• Comply and Compete—Model Management Best Practices (No. 55)
Conclusion
Top Related