© National Bank of Belgium. Failure Prediction Models: Disagreements, Performance, and Credit...


Failure Prediction Models: Disagreements, Performance, and Credit Quality

Janet MITCHELL and Patrick VAN ROYNational Bank of Belgium

“Small business banking and financing: a global perspective”

Cagliari, May 25th


Motivation

The paper explores empirically a number of comparative issues relating to models assigning failure predictions (credit scores or PDs) to non-listed firms.

Failure prediction models are important in Basel II (PDs key input for the calculation of regulatory capital under IRB approach).

Focus on four models: the National Bank of Belgium (NBB) bankruptcy prediction model, two vendor models (Model 1 and Model 2) and the Z-score (Altman).


Four main issues

Disagreements between models: do different models yield significantly different "rankings" for the same firm?

Model power: are some models better at differentiating between failing and non-failing firms?

Combining models: are combinations of models more powerful than single models?

Design of internal ratings systems: does model power change as the number of rating classes and the distribution of borrowers across classes vary?


Data and sample

40,000 small to medium-sized non-listed Belgian firms.

Inputs, statistical methods and calibration differ across models.

Data are obtained from the Belgian central balance sheet office and from one vendor.

Bankruptcy data is used to estimate 1-year and 5-year credit scores or PDs. The presentation focuses on 1-year failure predictions.


1-year bankruptcy rates (in %)

Risk class % firms NBB Model 1 Model 2 Z-score

1 01.4 0.00 0.00 0.00 0.00

2 21.5 0.00 0.01 0.06 0.05

3 21.5 0.09 0.06 0.11 0.25

4 18.8 0.22 0.16 0.20 0.45

5 22.0 0.34 0.43 0.57 0.40

6 11.6 1.44 2.00 1.12 1.28

7 03.3 7.85 5.52 6.85 5.46

Methodology

Output of each model (PDs or credit scores) is rank-ordered before being mapped to 1 of 7 risk classes based on the output of one vendor model:

low risk

high risk


Disagreements (1 vs. 4,5,6 or 7)

Percentage of class-1 firms (= lowest risk firms) of a given model classified in or above the median risk class (= class 4) by another model:

Class 1 Class 4,5,6 or 7

NBB Model 1 Model 2 Z-score

NBB - 01.4 36.7 54.2

Model 1 00.5 - 18.9 21.9

Model 2 34.3 29.4 - 42.3

Z-score 07.5 07.7 44.2 -


Disagreements (7 vs. 1,2,3 or 4)

Percentage of class-7 firms (= highest risk firms) of a given model classified in or below the median risk class (= class 4) by another model:

Class 7 Class 1,2,3 or 4

NBB Model 1 Model 2 Z-score

NBB - 16.3 14.5 16.4

Model 1 08.7 - 15.6 33.2

Model 2 18.1 19.0 - 29.3

Z-score 18.4 26.6 11.5 -


Model power: ROC curves

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Non-failing firms ordered by model score percentile

Percentage of

failing firms

Hypotheticalmodel

Randomchoice

type 1 error

1 type 2 error


ROC curves of the 4 models

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Non-failing firms ordered by model score percentile

Percentage of

failing firms

NBB

Model 1

Model 2

Z-score

Randomchoice


ROC areas of the 4 models

Model Area under the ROC curve

NBB 0.876

Model 1 0.868

Model 2 0.833

Z-score 0.779

Area of model with no discriminatory power = 0.5Area of model with acceptable discriminatory power > 0.7Area of model with perfect discriminatory power = 1.0


ROC areas of selected combinations

NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833;

Z-score (Z) = 0.779

Combination Min. Max. Median Mean

N - M1 0.878 0.898 0.908

N - M2 0.861 0.892 0.898

N - Z 0.854 0.855 0.880

N - M1 - M2 0.867 0.901 0.899 0.916

N - M1 - Z 0.861 0.886 0.894 0.911

N - M2 - Z 0.845 0.879 0.883 0.899

N - M1- M2 - Z 0.852 0.890 0.914 0.917


ROC areas of selected combinations

NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833;

Z-score (Z) = 0.779

Combination Min. Max. Median Mean

Z - N 0.854 0.855 0.880

Z - M1 0.847 0.871 0.890

Z - M2 0.808 0.858 0.855


ROC areas of 9 possible internal ratings systems (NBB model)

Number of

classes

Mapping of firms based on

Vendor model distribution

Moody's distribution

Equal distribution

7 0.876 0.874 0.858

10 0.883 0.882 0.873

17 0.887 0.885 0.883

Note: NBB continuous credit score has an ROC area of 0.889

0.018

0.010

0.0040.011 0.011 0.025


Conclusion

High disagreements rates between models: model choice can have a significant impact on loan pricing and origination decisions.

High power of each model: the definition of failure as well as the statitical method used by the models may not matter as much as one would have expected.

Larger differences between differing combinations of models than between differing internal rating systems.

© National Bank of Belgium. Failure Prediction Models: Disagreements, Performance, and Credit...

Documents

Transcript of © National Bank of Belgium. Failure Prediction Models: Disagreements, Performance, and Credit...