risk.validation.binomialTest

R2026b

Binomial test

Since R2025a

Syntax

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials)

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value)

[hBinTest,binOutput] = risk.validation.binomialTest(___)

Description

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials) returns the binomial test result or results, hBinTest, for a given set of probabilities, events, and trials. The output is 1 if the test rejects the null hypothesis at the 95% confidence level, or 0 otherwise. Probability contains numeric values that represent quantities such as probability of default (PD) estimates.

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value) specifies optional name-value arguments. For example, you can set a specific confidence level for the binomial test by using the ConfidenceLevel name-value argument.

[hBinTest,binOutput] = risk.validation.binomialTest(___) also returns a structure binOutput that contains the method used in the binomial test and a table of summary metrics. Specify binOutput as the second output argument with any of the input argument combinations in the previous syntaxes.

Examples

Apply Binomial Test on Probability of Default Data

Use the risk.validation.binomialTest function to determine whether the binomial test rejects the null hypothesis for a 0.95 confidence level in a probability of default (PD) data set. In this example, you use the credit validation data set, which includes a table, ScorecardValidationData, that contains PD values and their corresponding default status. Before you apply the test, it is common practice to:

• Group the probabilities by deciles.

• Compute the average probability of each group.

• Compute the total number of defaults and loans of each group.

After performing these steps, you then apply the test on each of the groups.

Load and display the credit validation data.

load CreditValidationData.mat
head(ScorecardValidationData)

    CreditScore      PD       Default
    ___________    _______    _______

      579.86       0.14182       0   
      563.65       0.17143       0   
      549.52       0.20106       0   
      546.25       0.20845       0   
      485.34       0.37991       0   
      482.07       0.39065       0   
      579.86       0.14182       1   
      451.73         0.494       0   

Group Probabilities by Deciles

Extract the variable PD from the table ScorecardValidationData and group the probabilities by using the risk.validation.groupNumberByQuantile function with the fully qualified namespace risk.validation. Specify deciles as the quantile type.

Probability = ScorecardValidationData.PD;
QuantileType = "deciles";
PDDecileNumber = risk.validation.groupNumberByQuantile(Probability,QuantileType);

Determine Average Probabilities by Group

Next, calculate the average probability for each group by using the groupsummary function.

PDAvgByDecile = groupsummary(Probability,PDDecileNumber,"mean");

Compute Total Defaults and Loans by Group

Extract the variable Default from the table ScorecardValidationData and use this variable as the default indicator. Then use the groupsummary function to compute the total number of defaults and loans for each group.

DefaultIndicator = ScorecardValidationData.Default;
[NumDefaultsByDecile,~,NumLoansByDecile] = groupsummary(DefaultIndicator,PDDecileNumber,"sum");

Apply Binomial Test

You can then apply the risk.validation.binomialTest function with the fully qualified namespace risk.validation for each group to see if the test rejects the null hypothesis. Use the average probabilities and the total number of defaults and loans as input arguments. Then, display a table with the results.

[hBinTest,binOutput] = risk.validation.binomialTest(PDAvgByDecile,NumDefaultsByDecile,NumLoansByDecile)

hBinTest = 10×1

     0
     0
     0
     0
     0
     0
     0
     0
     0
     0

binOutput = struct with fields:
     Method: "exact"
       Tail: "right"
    Results: [10×8 table]

disp(binOutput.Results)

    RejectBinTest     PValue     NumEvents    CriticalValue    ConfidenceLevel    NumTrials    Probability    ObservedProbability
    _____________    ________    _________    _____________    _______________    _________    ___________    ___________________

          0           0.21501        7              9               0.95             36          0.13683            0.19444      
          0          0.083864       11             12               0.95             36          0.19799            0.30556      
          0           0.51798        8             13               0.95             34          0.22656            0.23529      
          0           0.12711       10             12               0.95             27          0.25552            0.37037      
          0           0.12069       16             18               0.95             42           0.2868            0.38095      
          0           0.60973       13             19               0.95             41          0.32664            0.31707      
          0           0.92069       10             19               0.95             36          0.37627            0.27778      
          0           0.67016       11             17               0.95             29          0.40319            0.37931      
          0           0.58724       18             24               0.95             40          0.45525               0.45      
          0           0.55783       22             28               0.95             39          0.56233             0.5641      

Input Arguments

Probability — Probability values

numeric vector with values in the range (0,1)

Probability values, specified as a numeric vector with values in the range (0,1). Probability contains values that indicate quantities such as PD estimates.

NumEvents — Number of events observed

vector of nonnegative integers

Number of events observed, specified as a vector of nonnegative integers. For PD models, NumEvents contains the number of defaults observed.

NumTrials — Number of trials

vector of positive integers

Number of trials, specified as a vector of positive integers. Each element in NumTrials must be greater than or equal to the corresponding elements of NumEvents. For PD models, NumTrials contains the number of loans.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,ConfidenceLevel=0.99) specifies a confidence level of 0.99.

ConfidenceLevel — Confidence level

0.95 (default) | numeric scalar in the range (0,1)

Confidence level of the hypothesis test, specified as a numeric scalar in the range (0,1).

Method — Hypothesis test method

"exact" (default) | "approximate"

Hypothesis test method, specified as either "exact" or "approximate".

• "exact" — The function uses an exact probability based on a binomial distribution.

• "approximate" — The function uses an approximate test based on a normal distribution.

Data Types: string | char

Tail — Type of alternative hypothesis

"right" (default) | "left" | "both"

Since R2026a

Type of alternative hypothesis, specified as either "right", "left", or "both".

• "right" — Test against the alternative hypothesis that the probability is greater than Probability.

• "left" — Test against the alternative hypothesis that the probability is smaller than Probability.

• "both" — Test against the alternative hypothesis that the probability is not Probability.

Data Types: string | char

Output Arguments

hBinTest — Hypothesis test results

numeric vector containing values of 1 or 0

Hypothesis test results, returned as a numeric vector containing values of 1 or 0.

• A value of 1 rejects the null hypothesis at the specified confidence level.

• A value of 0 fails to reject the null hypothesis at the specified confidence level.

binOutput — Output metrics

structure

Output metrics, returned as a structure with the following fields:

• Method — Method used for the hypothesis test.

• Results — Table with columns:

  • RejectBinTest — Numeric value that indicates whether the null hypothesis was rejected. This column represents the same values as hBinTest.

  • PValue — Numeric value in the range [0,1] representing the p-value for the hypothesis test. A small value indicates that the null hypothesis might not be valid.

  • NumEvents — Numeric value representing the number of events observed. This column is the test statistic for the hypothesis test.

  • CriticalValue — Numeric value representing the minimum number of events at which the test rejects the null hypothesis for the given probability and confidence level.

  • ConfidenceLevel — Numeric value representing the confidence level for the hypothesis test.

  • NumTrials — Numeric value representing the number of trials.

  • Probability — Numeric value representing the probability values.

  • ObservedProbability — Numeric value representing the ratio of NumEvents to NumTrials.

More About

Binomial Test

The binomial test [1] requires you to input the predicted probability P, the number of trials N, and the number of events observed, E. The null hypothesis is that the probability P is a correct estimate of the event frequency. The alternative hypothesis for the test depends on the value of the Tail name-value argument. The test statistic for the binomial test is the number of events E.

The test has two versions: the exact method and the approximate method. In the exact method, the critical values for the left- and right-tailed tests are given by the following:

$$E_{\text{left}}^{*}=\max \left\{k:\underset{i=0}{\sum ^k}\left(\begin{array}{c}N\\ i\end{array}\right)P^i{\left(1-P\right)}^{N-i}\le \alpha \right\}\text{and}{E}_{\text{right}}^{*}=\min \left\{k:\underset{i=k}{\sum ^N}\left(\begin{array}{c}N\\ i\end{array}\right)P^i{\left(1-P\right)}^{N-i}\le \alpha \right\}\text{.}$$

where α = 1 − ConfidenceLevel is the significance level for the test. The software rejects the null hypothesis for the right-tailed test when the test statistic is larger than or equal to E^*_right. The software rejects the null hypothesis for the left-tailed test when the test statistic is smaller than or equal to E^*_left. To calculate the exact critical values E^*_left and E^*_right for the two-tailed test, the software sorts the possible outcomes in ascending order of their probabilities. Then, it calculates the integer T such that

$$\underset{j=1}{\sum ^T}{p}_j\le \alpha <\underset{j=1}{\sum ^{T+1}}{p}_j\text{,}$$

where p_j is the jth smallest probability. The critical values are given by the equations

$$E_{\text{left}}^{*}=\underset{j\le T}{\max }\left\{E_j^{*}:E_j^{*}<M\right\}\text{and}{E}_{\text{right}}^{*}=\underset{j\le T}{\max }\left\{E_j^{*}:E_j^{*}>M\right\}\text{,}$$

where E_j is the outcome corresponding to p_j, and M is the median of the binomial distribution given by N and P. The software rejects the null hypothesis when the test statistic is outside of the interval $$\left[E_{\text{left}}^{*}+1,E_{\text{right}}^{*}-1\right]$$.

In the approximate method, using the central limit theorem, the critical values for the left- and right-tailed tests are given by:

$$E_{\text{approx, left}}^{*}=NP+{\Phi }^{-1}\left(\alpha \right)\sqrt{NP\left(1-P\right)}\text{and}{E}_{\text{approx, right}}^{*}=NP+{\Phi }^{-1}\left(1-\alpha \right)\sqrt{NP\left(1-P\right)}\text{,}$$

where Φ is the standard normal cumulative distribution function. The critical values for the two-tailed test are given by

$$E_{\text{approx, left}}^{*}=NP+{\Phi }^{-1}\left(\frac{1}{2}\alpha \right)\sqrt{NP\left(1-P\right)}\text{and}{E}_{\text{approx, right}}^{*}=NP+{\Phi }^{-1}\left(1-\frac{1}{2}\alpha \right)\sqrt{NP\left(1-P\right)}\text{.}$$

If the inputs to risk.validation.binomialTest are vectors of size G, then G statistical tests are run and the function returns the results for each of these tests.

For credit risk applications, P is typically a PD, N is the number of loans, and E the number of defaults. The G groups could correspond to different credit ratings or deciles based on the PD values of a continuous PD model. For market risk applications, P could represent the complement of the VaR level, N could represent the number of days in a test window such as 250 trading days, and E could represent the number of VaR exceptions.

Alternative Functionality

You can use the varbacktest object function bin to perform a binomial test on a timeseries of portfolio outcomes. The bin function allows you to perform binomial tests on multiple timeseries with different VaR confidence levels.

References

[1] Basel Committee on Banking Supervision. “Studies on the Validation of Internal Rating Systems.” Working Paper 14. May, 2005. https://www.bis.org/publ/bcbs_wp14.htm.

Version History

Introduced in R2025a

R2026a: Tail name-value argument to specify type of alternative hypothesis

Specify the type of alternative hypothesis using the Tail name-value argument.