Summary of the impact

In response to the deficiencies in bank risk management revealed following the 2008 financial crisis, one of the mandated requirements under the Basel III regulatory framework is for banks to backtest the internal models they use to price their assets and to calculate how much capital they require should a counterparty default. Qiwei Yao worked with the Quantitative Analyst — Exposure team at Barclays Bank, which is responsible for constructing the Barclays Counterpart Credit Risk (CCR) backtesting methodology. They made use of several statistical methods from Yao's research to construct the newly developed backtesting methodology which is now in operation at Barclays Bank. This puts the CCR assessment and management at Barclays in line with the Basel III regulatory capital framework.

Underpinning research

Research Insights and Outputs: The Barclays CCR backtesting methodology, upon which Qiwei Yao's research had an impact, integrates a number of statistical methods, underpinned by four pieces of Yao's research. Each of these four research contributions arose as part of his long-term focus on statistical inference for time series.

In an effort to reveal dynamic structure beyond linear autocorrelation, Yao has made substantial advances in developing methods for modelling and forecasting the future conditional on current and past status, which reveals various interesting features that are relevant to but absent from conventional linear time series models. For example, prediction errors for the future depend on current position and errors are nonlinearly amplified over time. Yao's work in this area involves a number of co-authors. For the method used for the CCR project, of particular relevance was his joint work [1] published in 2001 with Zongwu Cai from the University of North Carolina at Charlotte and Wenyang Zhang, then a postdoctoral research officer at LSE and now Professor in the Department of Mathematics at the University of York. In terms of technical tools, Yao and his co-authors have developed several nonparametric and semiparametric methods, including a method in [1] for estimating conditional forecasters in the form of point forecasts, forecast sets and forecasting distributions. To assess the accuracy of those forecasters, some resampling techniques are used. A distinguishing feature is to make the resampling adaptive to the dependence in the data. This is achieved by either reproducing the dependence in the resampled data or resampling the pre-whitened data.

Yao's second underpinning research contribution in [2] in 2000 was joint work with two LSE Visiting Professors, Jianqing Fan and Peter Hall, on a project investigating the use of traditional Z-tests and t-tests and also bootstrap calibration in the context of simultaneous hypothesis testing. One of the interesting findings in [2] was that with bootstrap methods the number of simultaneous tests can be substantially larger than the sample size. In fact, log(v) can be as large as the square-root of the sample size, where v denotes the number of simultaneous tests. This justifies the use of bootstrap methods when traditional Z-tests and t-test are inapplicable.

In the third more recent contribution, Yao, jointly with Hongzhi An (Chinese Academy of Sciences), Da Huang (Yao's PhD student at the time) and Cun-Hui Zhang (Rutgers University), revisited the classic stepwise selection methods for regression in the setting of "large p and small n", see [3]. They showed that model selection consistency can be achieved by using stepwise selection coupled with appropriately modified information criteria. They also demonstrated by simulation that their method provides a much more robust performance than some popular procedures such as LASSO. This method provides a stable initial screening among a large number of trades for selecting a counterparty representative portfolio.

Finally the matching quantiles estimation method proposed in [4] directly resulted from solving the counterparty representative portfolio selection problem. A new measure and a new statistical test were also proposed in [4] to measure the goodness of match. It was joint work with Nikolaos Sgouropoulos at QA Exposure of Barclays Bank and Claudia Yastremiz of Bank of England.

Key Researcher: Professor Yao has been at LSE since 2000.

References to the research

Evidence of quality: (1) and (2) are in top, peer-reviewed journals.

Details of the impact

BACKGROUND

Basel III, developed by the Basel Committee on Banking Supervision and agreed in September 2010, is a set of comprehensive reform measures that puts in place a global regulatory standard on bank capital adequacy, stress testing and market liquidity. One of its mandatory requirements is for all banks to conduct counterparty credit risk (CCR) model backtesting. However, it leaves each bank to define its own backtesting methodology.

CCR backtesting is intended to ensure that the models provide more timely and accurate information on a bank's exposure to the risk caused by a counterparty by comparing the risk measures implied by the bank's pricing models with the realised exposure based on the traded prices. The main output of the backtesting is in the form of a "traffic light" system whereby: "green" signals that there is no evidence against the pricing models; "amber" signals that observed risk exposure is higher than that implied by the pricing models but still within an acceptable tolerance level; "red" indicates that the price models underestimate the risk and need to be re-calibrated and the capital requirement adjusted. Ultimately, backtesting results should demonstrate to the regulators (i.e. the Bank of England in UK) the soundness and conservativeness of the reported exposure to risk. Backtesting should also be able to identify where pricing models are overly-conservative.

At the invitation of Barclays Bank PLC, Yao has been participating in the CCR backtesting project undertaken by the Quantitative Analyst — Exposure Group at Barclays since January 2012. Yao was invited because of his relevant research expertise in handling dependence and nonstationarity in data and his considerable experience in inference with conditional distributions.

NATURE AND EXTENT OF THE IMPACT

Based on his previous work in [1], [2] and [3], and the newly developed method in [4], Yao proposed several key statistical methods that were used in the development of the backtesting methodology outlined in document [A]; see also [B] and [C]. The methodology has been approved by Barclays internal governance process and, from September 2013, it has been implemented as a part of business operations at Barclays. The outputs of the methodology are now being used by Barclays credit risk managers on a daily basis to control model risks. The new methodology improves substantially the CCR assessment and management at Barclays in the ways described below, and puts the practice at Barclays in line with the Basel III regulatory capital framework. The resulting improved information about the bank's exposure to risk mitigates potential future losses and thus also helps to stabilize the global financial market and protect economic stability and individual welfare.

DETAILS OF HOW THE RESEARCH UNDERPINNED THE BACKTESTING METHODOLOGY AND HOW RISK ASSESSMENT HAS BEEN IMPROVED

Yao contributed directly to Barclays backtesting methodology set out in [A] and [B] (see also [C]). Indeed, Yao wrote the first version of [B]. The key ways in which his research underpinned the methodology and the benefits arising from his research contributions can be summarised as follows.

1. Yao's research fed into a conditional counter (a new metric for backtesting) and a simulation-based testing method which result in more reliable information on risk exposure.

A "binomial counter" had been used by Barclays for analysing collateralized transactions for which the data from non-over-lapping time intervals can be treated as independent. A stratified version was introduced to deal with dependence in uncollateralized transactions. However, this simple approach, although approved by the Financial Services Authority, is inadequate where prices across different time horizons are dependent on each other — which is commonplace. Indeed, there hardly exists any effective metrics for backtesting with dependent data.

The conditional counter and simulation-based testing method, introduced on the basis of Yao's research, fill this gap. The conditional counter method specified in [A] and [B] applies a version of the nonparametric estimation method proposed in [1]. It effectively looks at the extreme values of conditional distributions instead of those of unconditional distributions. The simulation method is more generic. It can be used to test not only the extreme quantiles but also other features such as the whole distribution. It can also be easily extended to test the sensitivity to risk factors. This is significant because an important new requirement of Basel III is to test various features of the distribution.

A simulation-based testing procedure for calculating p-values using the bootstrap multiple comparison method of [2] is generic. It can be used for testing, for example, a pricing model in relation to an observed trade path over different time horizons based on any appropriate test statistic. It takes into account the non-stationarity and the dependencies among prices at different time horizons, or different price paths, in an automatic manner (see [A]). This enables Barclays to extract more reliable information on risk exposure in their daily operation. This test procedure can also be adapted to identify whether or not pricing models are overly-conservative, providing a sound scientific basis for Barclays to adjust is capital reserve.

The proposed simulation-based testing method based on bootstrap calibration represents the first such generic method to incorporate nonstationarity and dependence among different trades and/or different time horizons in an almost automatic manner. This represents a big step forward in the backtesting techniques used at Barclays.

2. Yao's research contributed some key steps to the methodology for selecting representative portfolios , which result in the Basel III standard being met more effectively.

Basel III allows banks to construct representative portfolios for each counterparty consisting of, for example, a subset of the trades between two banks. Banks are left to decide the number and trades to be included in the portfolio, but they have to justify their choices to their supervisors (the Bank of England in the UK). As the number of trades between two major banks can easily be in the order of tens thousands or more, a simple linear regression runs into the so-called "large p and small n" problem, even after the initial screening and categorisation. Furthermore, many trades are highly correlated in the sense that the sparse representation is certainly not unique. Hence some popular techniques such as LASSO or the Danzig algorithm are no longer applicable. The procedure adopted by Barclays uses the method of [3], i.e. a stepwise sweep coupled with the use of (modified) information criteria to form the candidate set. To construct a representative portfolio from the trades in the candidate set, the new Matching Quantiles Estimation (MQE) method proposed in [4] is now employed. Barclays has also adopted a new measure and a test proposed in [4] to check how well the distribution of a selected portfolio matches the target distribution at all levels simultaneously, as required by Basel III. This overcomes problems with existing estimation methods, such as quantile regression, which can only check the success of a match at one, or at most, a few fixed levels, resulting in a representative portfolio that falls short of the Basel III standard.

Construction of counterparty representative portfolios is a new mandated requirement under Basel III. Some key steps in the way that Barclays has formulated this construction in its new methodology are attributable to Yao's research.

Sources to corroborate the impact

[A] Barclays QA Exposure Analytics (2012). (This source is confidential. Please see [C])

[B] Barclays QA Exposure Analytics (2012). (This source is confidential. Please see [C])

[C] A letter from the Director of the Barclays QA Exposure Analytics on Yao's contribution on the CCR backtesting. This source is confidential.

Yao received the initial invitation from Barclays to participate this project for 3 months (January — March 2012). The invitation has been subsequently extended to December 2013.