Documentation

This is machine translation

Translated by
Mouseover text to see original. Click the button below to return to the English verison of the page.

To view all translated materals including this page, select Japan from the country navigator on the bottom of this page.

Overview of Expected Shortfall Backtesting

For VaR backtesting, the possibilities every day are two: either there is a VaR failure or not. If the VaR confidence level is 95%, VaR failures should happen approximately 5% of the time. To backtest VaR, you only need to know whether the VaR was exceeded (VaR failure) or not on each day of the test window and the VaR confidence level. Risk Management Toolbox™ VaR backtesting tools support “frequency” (assess the proportion of failures) and “independence” (assess independence across time) tests, and these tests work with the binary sequence of "failure" or "no-failure" results over the test window.

For expected shortfall (ES), the possibilities every day are infinite: The VaR may be exceeded by 1%, or by 10%, or by 150%, and so on. For example, there are three VaR failures in the following example:

On failure days, the VaR is exceeded on average by 39%, but the estimated ES exceeds VaR by an average of 27%. How can you tell if 39% is significantly larger than 27%? Knowing the VaR confidence level is not enough, you must also know how likely are the different exceedances over the VaR according to the VaR model. In other words, you need some distribution information about what happens beyond the VaR according to your model assumptions. For thin-tail VaR models, 39% vs. 27% may be a large difference. However, for a heavy-tail VaR model where a severity of twice the VaR has a non-trivial probability of happening, then 39% vs. 27% over the three failure dates may not be a red flag.

A key difference between VaR backtesting and ES backtesting is that most ES backtesting methods require information about the distribution of the returns on each day, or at least the distribution of the tails beyond the VaR. One exception is the “unconditional” test (see `unconditionalNormal` and `unconditionalT`) where you can get approximate test results without providing the distribution information. This is important in practice, because the “unconditional” test is much simpler to use and can be used in principle for any VaR or ES model. The trade-off is that the approximate results may be inaccurate, especially in borderline accept, or reject cases, or for certain types of distributions.

The toolbox supports the following tests for expected shortfall backtesting for table-based tests for the unconditional Acerbi-Szekely test using the `esbacktest` object:

The toolbox also supports the following simulation-based tests for expected shortfall backtesting using the `esbacktestbysim` object:

For the simulation-based tests, you must provide the model distribution information as part of the imputs to `esbacktestbysim`.

Conditional Test

The conditional test statistic is based on the conditional relationship

`$E{S}_{t}=-{E}_{t}\left[{X}_{t}|{X}_{t}<-Va{R}_{t}\right]$`

where

`X`t is the portfolio outcome, that is, the portfolio return or portfolio profit and loss for period t.

`VaR`t is the estimated VaR for period t.

`ES`t is the estimated expected shortfall for period t.

The number of failures is defined as

`$NumFailures=\sum _{t=1}^{N}{I}_{t}$`

where

`N` is the number of periods in the test window (t = `1`,…,`N`).

`I`t is the VaR failure indicator on period t with a value of 1 if `X`t < -VaR, and 0 otherwise.

The conditional test statistic is defined as

The conditional test has two parts. A VaR backtest must be run for the number of failures (`NumFailures`), and a standalone conditional test is performed for the conditional test statistic `Z`cond. The conditional test accepts the model only when both the VaR test and the standalone conditional test accept the model. For more information, see `conditional`.

Unconditional Test

The unconditional test statistic is based on the unconditional relationship,

`$E{S}_{t}=-{E}_{t}\left[\frac{{X}_{t}{I}_{t}}{{p}_{VaR}}\right]$`

where

`X`t is the portfolio outcome, that is, the portfolio return or portfolio profit and loss for period t.

`P`VaR is the probability of VaR failure defined as 1-VaR level.

`ES`t is the estimated expected shortfall for period t.

`I`t is the VaR failure indicator on period t with a value of 1 if `X`t < -VaR, and 0 otherwise.

The unconditonal test statistic is defined as

The critical values for the unconditional test statistic are stable across a range of distributions, which is the basis for the table-based tests. The `esbacktest` class runs the unconditional test against precomputed critical values under two distributional assumptions, namely, normal distribution (thin tails, see `unconditionalNormal`), and t distribution with 3 degrees of freedom (heavy tails, see `unconditionalT`).

Quantile Test

A sample estimator of the expected shortfall for a sample `Y`1,…,`Y`N is given by

`$\stackrel{⌢}{ES}\left(Y\right)=-\frac{1}{⌊N{p}_{VaR}⌋}\sum _{i=1}^{⌊N{p}_{VaR}⌋}{Y}_{\left[i\right]}$`

where

`N` is the number of periods in the test window (t = `1`,…,`N`).

`P`VaR is the probability of VaR failure defined as 1-VaR level.

`Y`1,…,`Y`N are the sorted sample values (from smallest to largest), and $⌊N{p}_{VaR}⌋$ is the largest integer less than or equal to `Np`VaR.

To compute the quantile test statistic, a sample of size `N` is created at each time t as follows. First, convert the portfolio outcomes to `X`t to ranks ${U}_{1}={P}_{1}\left({X}_{1}\right),...,{U}_{N}={P}_{N}\left({X}_{N}\right)$ using the cumulative distribution function `P`t. If the distribution assumptions are correct, the rank values `U`1,…,`U`N are uniformly distributed in the interval (0,1). Then at each time t:

• Invert the ranks U = (`U`1,…,`U`N) to get `N` quantiles ${P}_{t}^{-1}\left(U\right)=\left({P}_{t}^{-1}\left({U}_{1}\right),...,{P}_{t}^{-1}\left({U}_{N}\right)\right)$.

• Compute the sample estimator $\stackrel{⌢}{ES}\left({P}_{t}^{-1}\left(U\right)\right)$.

• Compute the expected value of the sample estimator

where `V` = (`V`1,…,`V`N is a sample of `N` independent uniform random variables in the interval (0,1). This can be computed anaytically.

The quantile test statistic is defined as

`${Z}_{quantile}=-\frac{1}{N}\sum _{t=1}^{N}\frac{\stackrel{⌢}{ES}\left({P}_{t}^{-1}\left(U\right)\right)}{E\left[\stackrel{⌢}{ES}\left({P}_{t}^{-1}\left(V\right)\right)\right]}+1$`

The denominator inside the sum can be computed analytically as

`$E\left[\stackrel{⌢}{ES}\left({P}_{t}^{-1}\left(V\right)\right)\right]=-\frac{N}{⌊{N}_{pVaR}⌋}{\int }_{0}^{1}{I}_{1-p}\left(N-⌊{N}_{pVaR}⌋,⌊{N}_{pVaR}⌋\right){P}_{t}^{-1}\left(p\right)dp$`
where `I`x(`z`,`w`) is the regularized incomplete beta function. For more information, see `betainc` and `quantile`.

References

[1] Basel Committee on Banking Supervision, Supervisory framework for the use of “backtesting” in conjunction with the internal models approach to market risk capital requirements. January 1996, http://www.bis.org/publ/bcbs22.htm.

[2] Acerbi, C., and B. Szekely. Backtesting Expected Shortfall. MSCI Inc. December, 2014.