Designing the Next Generation of Retail Credit Risk Models
Submitting Institution
University of EdinburghUnit of Assessment
Business and Management StudiesSummary Impact Type
EconomicResearch Subject Area(s)
Economics: Applied Economics, Econometrics
Commerce, Management, Tourism and Services: Banking, Finance and Investment
Summary of the impact
Research carried out by members of the Credit Research Centre (CRC) at
the University of Edinburgh has changed the way that credit risk modelling
teams in major multinational retail banks think about and model the
probability that an applicant will default on a loan. Such models are used
monthly to assess the risk associated with most of the 58 million credit
cards in the UK and hundreds of millions of credit cards elsewhere in the
world. The research has also changed the way banks model a crucial
parameter in the amount of capital they have to hold to comply with
international regulations, and has allayed their concerns about estimating
models based on only samples of previously accepted applicants.
Underpinning research
The CRC has hosted a programme of research carried out by Crook (1993-),
Bellotti (2006-), Banasik (1993-2010), Ansell (1997-), Andreeva (2005-),
Leow (2010-) and several PhD students (Andreeva, Ma, Kay, Moreira, Hanley,
Li, Osipenko). The specific projects highlighted here concern the way
banks assess the riskiness of making a loan. All of the research described
was carried out in Edinburgh. It covers three main areas.
1. The design of survival models of consumer credit default.
Banks have to assess the creditworthiness of (a) each applicant for
credit, to decide whether to grant credit, and of (b) those account
holders who already have revolving credit facilities. Typically, banks use
a parameterised cross sectional logistic regression model to make a
prediction of the probability of default within a defined time period
(e.g. 18 months) for each applicant. This is compared with a critical
value to decide if credit will be granted or limit increased. The research
[3.1] carried out since 1997 showed that a bank could predict the
probability of default not just over a predefined time period but
in any future period, so long as the parameters of the model
continued to represent behaviour. Research since 2006 [3.2] also showed
that one should include predicted values of macroeconomic indicators in
such a model to make the predicted probabilities of default in each month
depend on the expected future states of the macro-economy. This allows a
bank to predict the chances of a borrower defaulting in a month the bank
chooses given the bank's forecast about the macro-economy. It also allows
the bank to predict the default rate for an entire portfolio of loans for
a predicted state of the economy.
2. Reject inference. Banks develop credit scoring models using
samples of people they have lent to in the past. They wish to estimate a
model that represents the behaviour of any potential borrower. There are
statistical reasons why the former may not represent the latter. If the
difference is major, banks will be accepting the wrong applicants for
loans. Reject inference is a procedure whereby bank statisticians try to
"correct" models that have been parameterised on a sample of accepted
applicants to make the model "representative" of all applicants.
In a series of papers ( [3.3], [3.4] also Journal of the Operational
Research Society (2010), European Journal of Operational
Research (2007)), Crook and Banasik showed that (a) the "bias"
resulting from estimating a model based on only accepted applicants was
minimal and that known estimators made only a modest improvement to
predictive accuracy; (b) that techniques that were conventionally used by
statisticians in the industry had little effect unless a very large
proportion of loan applications were rejected; and (c) that when the
conventional industry method is applied to a survival model, again there
is little improvement.
3. Loss Given Default (LGD). The Basel II and III Accords require
that banks hold an amount of capital, mainly equity, equal to at least a
given percentage of risk weighted assets. The latter depend partly on the
proportion of a loan that is lost in the event of default (LGD). The
research [3.5] examined the predictive accuracy of a range of algorithms
and data transformations and, crucially, the empirical role for states of
the macro-economy to find which algorithm and which transformation led to
the most accurate predictions. The inclusion of macroeconomic variables
was crucial because it offered a new way of computing "stressed" levels of
LGD, which are what is required in the Accords.
References to the research
Research Grant:
March 2006 - February 2009 EPSRC: EP/D505380/1 (PI: D Hand, Imperial
College, Co-I: J
Crook (Edinburgh) and L Thomas (University of Southampton) £504,000 to
fund research into
risk management in the personal financial services sector.
Research Publications:
3.1 Banasik, J., Crook, J. & Thomas, L. (1999) `Not if but when will
borrowers default?' Journal of the Operational Research Society,
1185-90. Awarded 1999 Goodeve Medal by UK OR Society (DOI:
10.1057/palgrave.jors.2600851).
3.2 Bellotti, T. & Crook, J. (2009) `Credit scoring with
macroeconomic variables using survival analysis'. Journal of the
Operational Research Society, 60(12), 1699-1707 (DOI:
10.1057/jors.2008.130).
3.3 Crook, J., Banasik, J. & Thomas, L. (2004) `Does reject inference
really improve the performance of application scoring models?' Journal
of Banking and Finance, 28, 857-874 (DOI:
10.1016/j.jbankfin.2003.10.010).
3.4 Crook, J. Banasik, J. & Thomas, L. (2003) `Sample selection bias
in credit scoring models' Journal of the Operational Research Society,
54, 822-832 (DOI: 10.1057/palgrave.jors.2601578).
3.5 Bellotti, T. & Crook, J. (2010) `Loss given default models
incorporating macroeconomic variables for credit cards'. International
Journal of Forecasting (DOI: 10.1016/j.ijforecast.2010.08.005)
Details of the impact
Since the early 2000s, and especially since the crisis, when they were
found to be short of capital, banks across the world have been keen to
predict the impact of changes in the macro-economy on the future riskiness
of individual loans. Unfortunately their current models did not allow them
to do this. Our work showed how banks could incorporate macro-economic
factors into credit risk models.
Senior managers at banks and regulatory agencies across the UK and in
over 30 other countries were made aware of the research through four main
processes. The first was by giving invited presentations to private banks.
For example, the credit analytics team at Itau-Unibanc, Brazil - one of
the largest banks in the South American continent - invited Crook to give
a presentation on survival analysis methodologies for credit scoring to
its team in San Paulo; he was also invited to speak at the jointly created
Bank Z-CRC Innovation Forum, and Lloyds Bank invited him to explain the
application of survival models to its modelling teams. The second involved
presenting the results at public conferences, for example the biennial
Credit Scoring and Credit Control X, XI, and XII in Edinburgh 2007, 2009
and 2011 and the International Federation of Operational Research and
Management Science conference in Dallas 2010, which were attended by over
1,200 practitioners. Presentations were also made by invitation to a
public bank, the Federal Reserve Bank of Philadelphia. Thirdly, awareness
was raised by gaining significant BBC coverage which, although occurring
in 2007, led to subsequent impact in 2008 [see 5.1]. Fourthly, impact was
created through private conversations with influential members of
regulatory agencies, for example the Financial Services Authority.
Implementation of the findings of this research has had a direct impact
on the improvement in the competitiveness of banks, because the accuracy
of a bank's models is a source of competitive advantage. For this reason
banks are reluctant to reveal the full impact of our work. Nonetheless, a
range of impacts can be demonstrated.
1. Design of Survival Models. Prior to this research, banks were
unaware of and so did not use survival models to assess credit risk. The
impact of the research was such that it transformed the way credit risk
model teams in banks think about both what they are trying to model and
the way they do it. This research was vitally significant for several
reasons. It meant a bank could change the horizon over which it required a
probability of default from 18 months to any period it wished, without
re-estimating the model and it could weight scheduled payment amounts in
each month by the probability of default to compute expected
revenue from a loan in each month. Incorporating expected costs would
allow a bank to compute the expected profits from making a loan in each
month and so did not have to rely on computing just the probability of
default over a fixed horizon. The research also meant a bank could compute
a probability of default that was insensitive to the macro economy (by
fixing the values of the macroeconomic factors and the behavioural
factors) as required by the Basel Accords. A bank could also compare the
expected profits with the predicted probability of default over a given
horizon and so make a loan based on both expected profit and predicted
risk. As a result of our research many multinational banks can predict the
probability of default in a chosen future time period rather than over a
fixed (at the time of modelling) time period, and they can include
macroeconomic variables in such models. As the testimonies below
demonstrate, the research has had a direct impact on choices made by banks
and credit scoring institutions by demonstrating the superiority of using
survival (sometimes called "duration") models rather than cross-sectional
models.
"The work you did in 1999 was a useful case study to illustrate its
feasibility. Typically the approach is used mainly for credit risk
modelling for loans and mortgages where there is a fixed repayment
schedule." [5.2] (3 June 2013)
"[Quotation removed]". [5.3]
"[Quotation removed]". [5.4], referring to Banasik et al (1999)
and Bellotti and Crook (2009)
"...we did use survival analysis (one of my staff constructed the
model) along the lines of the paper [3.1 above] (a proportional odds
model if I remember correctly). I note that without the preliminary CRC
paper we would probably not have had the confidence to do this, as we
needed to quote evidence that the model was at least as powerful as a
combination of logistic regression and vintage curves" [5.5]
Given the secrecy surrounding the use of techniques by banks these
commendations are especially positive.
2. Reject Inference. The nature of the impact was to increase the
understanding on the part of risk modelling teams of what exactly the
statistical problems are when they estimate a model based only on a sample
of accepted applicants. This has occurred since the results were first
made public in 2001. Thus [5.3] writes:
"[Quotation removed]"
And [5.5] writes:
"In all the scorecard constructions that I have undertaken or
supervised for banks and building societies these papers [3.3 and 3.4
above] have changed the way that I approached the `Rejected Applicant'
problem".
3. Loss Given Default. The Basel II Accord requires banks to use
a level of LGD that would occur when there is an economic downturn. By
providing confirmatory evidence of the robustness of a two step approach,
this research has led to a consistent approach across many banks. The
research impacts on banks in two ways. First it shows a comparison of the
accuracy of methods for predicting LGD in the context of significant
secrecy. That is, banks do not reveal their methods publicly and so cannot
compare the accuracy of their methods with the accuracy obtained by their
competitors. The paper provides a benchmark. Second, by including
macroeconomic variables into the model, stressed levels of LGD can be
directly computed by choosing hypothetical `stressed' levels of the
macroeconomic variables and substituting them into the model.
"In agreement with your approach many retail banks apply a two stage
LGD model as you described. Confirmation that this was a robust approach
was helpful in bringing about [a] reasonably standard approach across
many banks". [5.2]
Because of its awareness of these research projects a multinational bank
asked the CRC to undertake consultancy projects that directly altered
decisions that the bank made (details confidential and covered by an NDA,
but related to portfolios of billions of pounds sterling).
Sources to corroborate the impact
5.1. BBC website:
http://news.bbc.co.uk/1/hi/scotland/edinburgh_and_east/7087028.stm
(or http://tinyurl.com/pj8e2xc )
(Corroborates the process described by which impact is created.)
Individual users/beneficiaries who could be contacted by the REF team to
corroborate claims:
5.2. [text removed for publication]) (Corroborates the effects of research on survival
analysis that it altered the way banks think about modelling default. Also
corroborates effects of research on LGD on bank modelling — [text
removed for publication].)
5.3. [text removed for publication] (Corroborates the effects of research on survival
analysis that it altered the way banks think about modelling default. Also
corroborates effects of research on reject inference as influencing the
way consultancies implement reject inference -[text removed for publication].)
5.4. [text removed for publication] (Corroborates the effects of research on survival
analysis that it altered the way a major consultancy think about modelling
default -[text removed for publication].)
5.5. Principal, Avenir Risk (Corroborates the effects of research on
survival modelling on bank modelling — [text removed for publication].)