Main Content

modelAccuracyPlot

Scatter plot of predicted and observed EADs

Description

example

modelAccuracyPlot(eadModel,data) returns a scatter plot of observed vs. predicted exposure at default (EAD) data with a linear fit. modelAccuracyPlot supports comparison against a reference model. By default, modelAccuracyPlot plots in the EAD scale.

example

modelAccuracyPlot(___,Name=Value) specifies options using one or more name-value arguments in addition to the input arguments in the previous syntax. You can use the ModelLevel name-value argument to compute metrics using the underlying model's transformed scale.

example

h = modelAccuracyPlot(ax,___,Name=Value) specifies options using one or more name-value arguments in addition to the input arguments in the previous syntax and returns the figure handle h.

Examples

collapse all

This example shows how to use fitEADModel to create a Tobit model and then use modelAccuracyPlot to generate a scatter plot for predicted and observed EADs.

Load EAD Data

Load the EAD data.

load EADData.mat
head(EADData)
    UtilizationRate    Age     Marriage        Limit         Drawn          EAD    
    _______________    ___    ___________    __________    __________    __________

        0.24359        25     not married         44776         10907         44740
        0.96946        44     not married    2.1405e+05    2.0751e+05         40678
              0        40     married        1.6581e+05             0    1.6567e+05
        0.53242        38     not married    1.7375e+05         92506        1593.5
         0.2583        30     not married         26258        6782.5        54.175
        0.17039        54     married        1.7357e+05         29575        576.69
        0.18586        27     not married         19590          3641        998.49
        0.85372        42     not married    2.0712e+05    1.7682e+05    1.6454e+05
rng('default');
NumObs = height(EADData);
c = cvpartition(NumObs,'HoldOut',0.4);
TrainingInd = training(c);
TestInd = test(c);

Select Model Type

Select a model type for Tobit or Regression.

ModelType = "Tobit";

Select Conversion Measure

Select a conversion measure for the EAD response values.

ConversionMeasure = "CCF";

Create Tobit EAD Model

Use fitEADModel to create a Tobit model using the TrainingInd data.

eadModel = fitEADModel(EADData(TrainingInd,:),ModelType,PredictorVars={'UtilizationRate','Age','Marriage'}, ...
    ConversionMeasure=ConversionMeasure,DrawnVar="Drawn",LimitVar="Limit",ResponseVar="EAD");
disp(eadModel);
  Tobit with properties:

        CensoringSide: "right"
            LeftLimit: NaN
           RightLimit: 1
              ModelID: "Tobit"
          Description: ""
      UnderlyingModel: [1x1 risk.internal.credit.TobitModel]
        PredictorVars: ["UtilizationRate"    "Age"    "Marriage"]
          ResponseVar: "EAD"
             LimitVar: "Limit"
             DrawnVar: "Drawn"
    ConversionMeasure: "ccf"

Display the underlying model. The underlying model's response variable is the transformation of the EAD response data. Use the 'LimitVar' and 'DrawnVar' name-value arguments to modify the transformation.

disp(eadModel.UnderlyingModel);
Tobit regression model, right-censored:
     EAD_ccf = min(Y*,1)
     Y* ~ 1 + UtilizationRate + Age + Marriage

Estimated coefficients:
                             Estimate        SE        tStat       pValue  
                            __________    ________    _______    __________

    (Intercept)                0.51514     0.11401     4.5184    6.5073e-06
    UtilizationRate            -1.5872    0.086106    -18.433             0
    Age                     -0.0058078    0.002712    -2.1416       0.03232
    Marriage_not married     0.0077853    0.074158    0.10498        0.9164
    (Sigma)                     1.4689    0.020619     71.241             0

Number of observations: 2627
Number of left-censored observations: 0
Number of uncensored observations: 2626
Number of right-censored observations: 1
Log-likelihood: -6311.87

Predict EAD

EAD prediction operates on the underlying compact statistical model and then transforms the predicted values back to the EAD scale. You can specify the predict function with different options for the 'ModelLevel' name-value argument.

predictedEAD = predict(eadModel,EADData(TestInd,:),ModelLevel="ead");
predictedConversion = predict(eadModel,EADData(TestInd,:),ModelLevel="ConversionMeasure");

Validate EAD Model

For model validation, use modelDiscrimination, modelDiscriminationPlot, modelAccuracy, and modelAccuracyPlot.

Use modelDiscrimination and then modelDiscriminationPlot to plot the ROC curve.

ModelLevel = "ead";

[DiscMeasure1,DiscData1] = modelDiscrimination(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel);
modelDiscriminationPlot(eadModel,EADData(TestInd, :),ModelLevel=ModelLevel,SegmentBy="Marriage");

Figure contains an axes object. The axes object with title EAD ROC Segmented by Marriage contains 2 objects of type line. These objects represent Tobit, married, AUROC = 0.79769, Tobit, not married, AUROC = 0.79825.

Use modelAccuracy and then modelAccuracyPlot to show a scatter plot of the predictions.

YData = "Observed";

[AccMeasure1,AccData1] = modelAccuracy(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel);
modelAccuracyPlot(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel,YData=YData);

Figure contains an axes object. The axes object with title Scatter Tobit, R-Squared: 0.33743 contains 2 objects of type scatter, line. These objects represent Data, Fit.

This example shows how to use fitEADModel to create a Beta model and then use modelAccuracyPlot to generate a scatter plot for predicted and observed EADs.

Load EAD Data

Load the EAD data.

load EADData.mat
head(EADData)
    UtilizationRate    Age     Marriage        Limit         Drawn          EAD    
    _______________    ___    ___________    __________    __________    __________

        0.24359        25     not married         44776         10907         44740
        0.96946        44     not married    2.1405e+05    2.0751e+05         40678
              0        40     married        1.6581e+05             0    1.6567e+05
        0.53242        38     not married    1.7375e+05         92506        1593.5
         0.2583        30     not married         26258        6782.5        54.175
        0.17039        54     married        1.7357e+05         29575        576.69
        0.18586        27     not married         19590          3641        998.49
        0.85372        42     not married    2.0712e+05    1.7682e+05    1.6454e+05
rng('default');
NumObs = height(EADData);
c = cvpartition(NumObs,'HoldOut',0.4);
TrainingInd = training(c);
TestInd = test(c);

Select Model Type

Select a model type for Beta.

ModelType = "Beta";

Select Conversion Measure

Select a conversion measure for the EAD response values.

ConversionMeasure = "LCF";

Create Beta EAD Model

Use fitEADModel to create a Beta model using the TrainingInd data.

eadModel = fitEADModel(EADData(TrainingInd,:),ModelType,PredictorVars={'UtilizationRate','Age','Marriage'}, ...
    ConversionMeasure=ConversionMeasure,DrawnVar="Drawn",LimitVar="Limit",ResponseVar="EAD");
disp(eadModel);
  Beta with properties:

    BoundaryTolerance: 1.0000e-07
              ModelID: "Beta"
          Description: ""
      UnderlyingModel: [1x1 risk.internal.credit.BetaModel]
        PredictorVars: ["UtilizationRate"    "Age"    "Marriage"]
          ResponseVar: "EAD"
             LimitVar: "Limit"
             DrawnVar: "Drawn"
    ConversionMeasure: "lcf"

Display the underlying model. The underlying model's response variable is the transformation of the EAD response data. Use the 'LimitVar' and 'DrawnVar' name-value arguments to modify the transformation.

disp(eadModel.UnderlyingModel);
Beta regression model:
     logit(EAD_lcf) ~ 1_mu + UtilizationRate_mu + Age_mu + Marriage_mu
     log(EAD_lcf) ~ 1_phi + UtilizationRate_phi + Age_phi + Marriage_phi

Estimated coefficients:
                                 Estimate        SE         tStat        pValue  
                                __________    _________    ________    __________

    (Intercept)_mu                -0.65566      0.11484     -5.7093    1.2616e-08
    UtilizationRate_mu              1.7014     0.078094      21.787             0
    Age_mu                      -0.0055901    0.0027603     -2.0252       0.04295
    Marriage_not married_mu      -0.012576     0.052098     -0.2414       0.80927
    (Intercept)_phi               -0.50131     0.094625     -5.2979    1.2687e-07
    UtilizationRate_phi            0.39731     0.066707       5.956    2.9305e-09
    Age_phi                      -0.001167    0.0023161    -0.50388       0.61439
    Marriage_not married_phi     -0.013275     0.042627    -0.31143        0.7555

Number of observations: 2627
Log-likelihood: -3140.21

Predict EAD

EAD prediction operates on the underlying compact statistical model and then transforms the predicted values back to the EAD scale. You can specify the predict function with different options for the 'ModelLevel' name-value argument.

predictedEAD = predict(eadModel,EADData(TestInd,:),ModelLevel="ead");
predictedConversion = predict(eadModel,EADData(TestInd,:),ModelLevel="ConversionMeasure");

Validate EAD Model

For model validation, use modelDiscrimination, modelDiscriminationPlot, modelAccuracy, and modelAccuracyPlot.

Use modelDiscrimination and then modelDiscriminationPlot to plot the ROC curve.

ModelLevel = "ead";

[DiscMeasure1,DiscData1] = modelDiscrimination(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel);
modelDiscriminationPlot(eadModel,EADData(TestInd, :),ModelLevel=ModelLevel,SegmentBy="Marriage");

Figure contains an axes object. The axes object with title EAD ROC Segmented by Marriage contains 2 objects of type line. These objects represent Beta, married, AUROC = 0.80597, Beta, not married, AUROC = 0.81734.

Use modelAccuracy and then modelAccuracyPlot to show a scatter plot of the predictions.

YData = "Observed";

[AccMeasure1,AccData1] = modelAccuracy(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel);
modelAccuracyPlot(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel,YData=YData);

Figure contains an axes object. The axes object with title Scatter Beta, R-Squared: 0.38655 contains 2 objects of type scatter, line. These objects represent Data, Fit.

Input Arguments

collapse all

Exposure at default model, specified as a previously created Regression, Tobit, or Beta object using fitEADModel.

Data Types: object

Data, specified as a NumRows-by-NumCols table with predictor and response values. The variable names and data types must be consistent with the underlying model.

Data Types: table

(Optional) Valid axis object, specified as an ax object that is created using axes. The plot will be created in the axes specified by the optional ax argument instead of in the current axes (gca). The optional argument ax must precede any of the input argument combinations.

Data Types: object

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: modelAccuracyPlot(eadModel,data(TestInd,:),DataID=Testing,XData='residuals',YData='residuals')

Data set identifier, specified DataID and a character vector or string. The DataID is included in the output for reporting purposes.

Data Types: char | string

Model level, specified as ModelLevel and a character vector or string.

Note

Regression models support all three model levels, but a Tobit or Beta model supports model levels only for "ead" and "conversionMeasure".

Data Types: char | string

EAD values predicted for data by the reference model, specified as ReferenceEAD and a NumRows-by-1 numeric vector. The scatter plot output is plotted for both the eadModel object and the reference model.

Data Types: double

Identifier for the reference model, specified as ReferenceID and a character vector or string. ReferenceID is used in the scatter plot output for reporting purposes.

Data Types: char | string

Data to plot on x-axis, specified as XData and a character vector or string for one of the following:

  • 'predicted' — Plot the predicted EAD values in the x-axis.

  • 'observed' — Plot the observed EAD values in the x-axis.

  • 'residuals' — Plot the residuals in the x-axis.

  • VariableName — Use the name of the variable in the data input, not necessarily a model variable, to plot in the x-axis.

Data Types: char | string

Data to plot on y-axis, specified as YData and a character vector or string for one of the following:

  • 'predicted' — Plot the predicted EAD values in the y-axis.

  • 'observed' — Plot the observed EAD values in the y-axis.

  • 'residuals' — Plot the residuals in the y-axis.

Data Types: char | string

Output Arguments

collapse all

Figure handle for the scatter and line objects, returned as handle object.

More About

collapse all

Model Accuracy Plot

The modelAccuracyPlot function returns a scatter plot of observed vs. predicted loss given default (EAD) data with a linear fit and reports the R-square of the linear fit.

The XData name-value pair argument allows you to change the x values on the plot. By default, predicted EAD values are plotted in the x-axis, but predicted EAD values, residuals, or any variable in the data input, not necessarily a model variable, can be used as x values. If the selected XData is a categorical variable, a swarm chart is used. For more information, see swarmchart.

The YData name-value pair argument allows users to change the y values on the plot. By default, observed EAD values are plotted in the y-axis, but predicted EAD values or residuals can also be used as y values. YData does not support table variables.

The linear fit and reported R-squared value always correspond to the linear regression model with the plotted y values as response and the plotted x values as the only predictor.

References

[1] Baesens, Bart, Daniel Roesch, and Harald Scheule. Credit Risk Analytics: Measurement Techniques, Applications, and Examples in SAS. Wiley, 2016.

[2] Bellini, Tiziano. IFRS 9 and CECL Credit Risk Modelling and Validation: A Practical Guide with Examples Worked in R and SAS. San Diego, CA: Elsevier, 2019.

[3] Brown, Iain. Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Applications. SAS Institute, 2014.

[4] Roesch, Daniel and Harald Scheule. Deep Credit Risk. Independently published, 2020.

Version History

Introduced in R2021b

expand all