fitSVMPosterior

Fit posterior probabilities

Syntax

  • ScoreSVMModel = fitSVMPosterior(SVMModel) example
  • ScoreSVMModel = fitSVMPosterior(SVMModel,Name,Value) example
  • ScoreSVMModel = fitSVMPosterior(SVMModel,X,Y) example
  • [ScoreSVMModel,ScoreTransform] = fitSVMPosterior(___) example

Description

example

ScoreSVMModel = fitSVMPosterior(SVMModel) returns ScoreSVMModel, which is a trained, support vector machine (SVM) classifier containing the optimal score-to-posterior-probability transformation function for two-class learning.

The software fits the appropriate score-to-posterior-probability transformation function using the SVM classifierSVMModel, and by cross validation using the stored predictor data (SVMModel.X) and the class labels (SVMModel.Y). The transformation function computes the posterior probability that an observation is classified into the positive class (SVMModel.Classnames(2)).

  • If the classes are inseparable, then the transformation function is the sigmoid function.

  • If the classes are perfectly separable, the transformation function is the step function.

  • In two-class learning, if one of the two classes has a relative frequency of 0, then the transformation function is the constant function. fitSVMPosterior is not appropriate for one-class learning.

  • If SVMModel is a ClassificationSVM classifier, then the software estimates the optimal transformation function by 10-fold cross validation as outlined in [1]. Otherwise, SVMModel must be a ClassificationPartitionedModel classifier. SVMModel specifies the cross-validation method.

  • The software stores the optimal transformation function in ScoreSVMModel.ScoreTransform.

example

ScoreSVMModel = fitSVMPosterior(SVMModel,Name,Value) uses additional options specified by one or more Name,Value pair arguments provided SVMModel is a ClassificationSVM classifier.

For example, you can specify the number of folds to use in k-fold cross validation.

example

ScoreSVMModel = fitSVMPosterior(SVMModel,X,Y) returns a trained, support vector classifier containing the transformation function from the trained, compact SVM classifier SVMModel. The software estimates the score transformation function using predictor data X and class labels Y.

example

[ScoreSVMModel,ScoreTransform] = fitSVMPosterior(___) additionally returns the transformation function parameters (ScoreTransform) using any of the input arguments in the previous syntaxes.

Examples

expand all

Fit the Score-to-Posterior Probability Function for Separable Classes

Load Fisher's iris data set. Train the classifier using the petal lengths and widths, and remove the virginica species from the data.

load fisheriris
classKeep = ~strcmp(species,'virginica');
X = meas(classKeep,3:4);
y = species(classKeep);

gscatter(X(:,1),X(:,2),y);
title('Scatter Diagram of Iris Measurements')
xlabel('Petal length')
ylabel('Petal width')
legend('Setosa','Versicolor')

The classes are perfectly separable. Therefore, the score transformation function is a step function.

Train an SVM classifier using the data. Cross validate the classifer using 10-fold cross validation (the default).

rng(1);
CVSVMModel = fitcsvm(X,y,'CrossVal','on');

CVSVMModel is a trained ClassificationPartitionedModel SVM classifier.

Estimate the step function that transforms scores to posterior probabilities.

[ScoreCVSVMModel,ScoreParameters] = fitSVMPosterior(CVSVMModel);
Warning: Classes are perfectly separated. The optimal score-to-posterior
transformation is a step function. 

fitSVMPosterior does the following:

  • Uses the data that the software stored in CVSVMModel to fit the transformation function

  • Warns whenever the classes are separable

  • Stores the step function in ScoreCSVMModel.ScoreTransform

Display the score function type and its parameter values.

ScoreParameters
ScoreParameters = 

                        Type: 'step'
                  LowerBound: -0.8431
                  UpperBound: 0.6897
    PositiveClassProbability: 0.5000

ScoreParameters is a structure array with four fields:

  • The score transformation function type (Type)

  • The score corresponding to the negative class boundary (LowerBound)

  • The score corresponding to the positive class boundary (UpperBound)

  • The positive class probability (PositiveClassProbability)

Since the classes are separable, the step function transforms the score to either 0 or 1, which is the posterior probability that an observation is a versicolor iris.

Fit the Score-to-Posterior Probability Function for Inseparable Classes

Load the ionosphere data set.

load ionosphere

The classes of this data set are not separable.

Train an SVM classifier. Cross validate using 10-fold cross validation (the default). It is good practice to standardize the predictors and specify the class order.

rng(1) % For reproducibility
CVSVMModel = fitcsvm(X,Y,'ClassNames',{'b','g'},'Standardize',true,...
    'CrossVal','on');
ScoreTransform = CVSVMModel.ScoreTransform
ScoreTransform =

none

CVSVMModel is a trained ClassificationPartitionedModel SVM classifier. The positive class is 'g'. The ScoreTransform property is none.

Estimate the optimal score function for mapping observation scores to posterior probabilities of an observation being classified as 'g'.

[ScoreCVSVMModel,ScoreParameters] = fitSVMPosterior(CVSVMModel);
ScoreTransform = ScoreCVSVMModel.ScoreTransform
ScoreParameters
ScoreTransform =

@(S)sigmoid(S,-9.482017e-01,-1.218360e-01)


ScoreParameters = 

         Type: 'sigmoid'
        Slope: -0.9482
    Intercept: -0.1218

ScoreTransform is the optimal score transform function. ScoreParameters contains the score transformation function, slope estimate, and the intercept estimate.

You can estimate test-sample, posterior probabilities by passing ScoreCVSVMModel to kfoldPredict.

Estimate Posterior Probabilities for Test Samples

Estimate positive class posterior probabilities for the test set of an SVM algorithm.

Load the ionosphere data set.

load ionosphere

Train an SVM classifier. Specify a 20% holdout sample. It is good practice to standardize the predictors and specify the class order.

rng(1) % For reproducibility
CVSVMModel = fitcsvm(X,Y,'Holdout',0.2,'Standardize',true,...
    'ClassNames',{'b','g'});

CVSVMModel is a trained ClassificationPartitionedModel cross-validated classifier.

Estimate the optimal score function for mapping observation scores to posterior probabilities of an observation being classified as 'g'.

ScoreCVSVMModel = fitSVMPosterior(CVSVMModel);

ScoreSVMModel is a trained ClassificationPartitionedModel cross-validated classifier containing the optimal score transformation function estimated from the training data.

Estimate the out-of-sample positive class posterior probabilities. Display the results for the first 10 out-of-sample observations.

[~,OOSPostProbs] = kfoldPredict(ScoreCVSVMModel);
indx = ~isnan(OOSPostProbs(:,2));
hoObs = find(indx); % Holdout observation numbers
OOSPostProbs = [hoObs, OOSPostProbs(indx,2)];
table(OOSPostProbs(1:10,1),OOSPostProbs(1:10,2),...
    'VariableNames',{'ObservationIndex','PosteriorProbability'})
ans = 

    ObservationIndex    PosteriorProbability
    ________________    ____________________

     6                    0.17379           
     7                    0.89639           
     8                  0.0076593           
     9                    0.91603           
    16                   0.026714           
    22                  4.607e-06           
    23                     0.9024           
    24                  2.413e-06           
    38                  0.0004266           
    41                    0.86427           

Input Arguments

expand all

SVMModel — Trained SVM classifierClassificationSVM classifier | CompactClassificationSVM classifier | ClassificationPartitionedModel classifier

Trained SVM classifier, specified as a ClassificationSVM, CompactClassificationSVM, or ClassificationPartitionedModel classifier.

If SVMModel is a ClassificationSVM classifier, then you can set optional name-value pair arguments.

If SVMModel is a CompactClassificationSVM classifier, then you must input predictor data X and class labels Y.

X — Predictor datamatrix

Predictor data used to estimate the score-to-posterior-probability transformation function, specified as a matrix.

Each row of X corresponds to one observation (also known as an instance or example), and each column corresponds to one variable (also known as a feature).

The length of Y and the number of rows of X must be equal.

If you set 'Standardize',true in fitcsvm to train SVMModel, then the software standardizes the columns of X using the corresponding means in SVMModel.Mu and standard deviations in SVMModel.Sigma. If the software fits the transformation-function parameter estimates using standardized data, then the estimates might differ from estimation without standardized data.

Data Types: double | single

Y — Class labelscategorical array | character array | logical vector | vector of numeric values | cell array of strings

Class labels used to estimate the score-to-posterior-probability transformation function, specified as a categorical or character array, logical or numeric vector, or cell array of strings.

If Y is a character array, then each element must correspond to one class label.

The length of Y and the number of rows of X must be equal.

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside single quotes (' '). You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: 'KFold',8 performs 8-fold cross validation when SVMModel is a ClassificationSVM classifier.

'CVPartition' — Cross-validation partition[] (default) | cvpartition partition

Cross-validation partition used to compute the transformation function, specified as the comma-separated pair consisting of 'CVPartition' and a cvpartition partition as created by cvpartition. You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

crossval splits the data into subsets using cvpartition.

'Holdout' — Fraction of data for holdout validationscalar value in the range (0,1)

Fraction of data for holdout validation used to compute the transformation function, specified as the comma-separated pair consisting of 'Holdout' and a scalar value in the range (0,1). Holdout validation tests the specified fraction of the data, and uses the remaining data for training.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'Holdout',0.1

Data Types: double | single

'KFold' — Number of folds10 (default) | positive integer value

Number of folds to use when computing the transformation function, specified as the comma-separated pair consisting of 'KFold' and a positive integer value.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'KFold',8

Data Types: single | double

'Leaveout' — Leave-one-out cross-validation flag'off' (default) | 'on'

Leave-one-out cross-validation flag indicating whether to use leave-one-out cross validation to compute the transformation function, specified as the comma-separated pair consisting of 'Leaveout' and 'on' or 'off'. Use leave-one-out cross validation by using 'on'.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'Leaveout','on'

Output Arguments

expand all

ScoreSVMModel — Trained SVM classifierClassificationSVM classifier | CompactClassificationSVM classifier | ClassificationPartitionedModel classifier

Trained SVM classifier containing the estimated score transformation function, returned as a ClassificationSVM, CompactClassificationSVM, or ClassificationPartitionedModel classifier.

The ScoreSVMModel classifier type is the same as the SVMModel classifier type.

To estimate posterior probabilities, pass ScoreSVMModel and predictor data to predict. If you set 'Standardize',true in fitcsvm to train SVMModel, then predict standardizes the columns of X using the corresponding means in SVMModel.Mu and standard deviations in SVMModel.Sigma.

ScoreTransform — Optimal score-to-posterior-probability transformation function parametersstructure array

Optimal score-to-posterior-probability transformation function parameters, specified as a structure array. If field Type is:

  • sigmoid, then ScoreTransform has these fields:

    • Slope — The value of A in the sigmoid function

    • Intercept — The value of B in the sigmoid function

  • step, then ScoreTransform has these fields:

    • PositiveClassProbability: the value of π in the step function. π represents:

      • The probability that an observation is in the positive class.

      • The posterior probability that a score is in the interval (LowerBound,UpperBound).

    • LowerBound: the value maxyn=1sn in the step function. It represents the lower bound of the interval that assigns the posterior probability of being in the positive class PositiveClassProbability to scores. Any observation with a score less than LowerBound has posterior probability of being the positive class 0.

    • UpperBound: the value minyn=+1sn in the step function. It represents the upper bound of the interval that assigns the posterior probability of being in the positive class PositiveClassProbability. Any observation with a score greater than UpperBound has posterior probability of being the positive class 1.

  • constant, then ScoreTransform.PredictedClass contains the name of the class prediction.

    This result is the same as SVMModel.ClassNames. The posterior probability of an observation being in ScoreTransform.PredictedClass is always 1.

    More About

    expand all

    Sigmoid Function

    The sigmoid function that maps score sj corresponding to observation j to the positive class posterior probability is

    P(sj)=11+exp(Asj+B).

    If the output argument ScoreTransform.Type is sigmoid, then parameters A and B correspond to the fields Scale and Intercept of ScoreTransform, respectively.

    Step Function

    The step function that maps score sj corresponding to observation j to the positive class posterior probability is

    P(sj)={0;s<maxyk=1skπ;maxyk=1sksjminyk=+1sk1;sj>minyk=+1sk,

    where:

    • sj the score of observation j.

    • +1 and –1 denote the positive and negative classes, respectively.

    • π is the prior probability that an observation is in the positive class.

    If the output argument ScoreTransform.Type is step, then the quantities maxyk=1sk and minyk=+1skcorrespond to the fields LowerBound and UpperBound of ScoreTransform, respectively.

    Constant Function

    The constant function maps all scores in a sample to posterior probabilities 1 or 0.

    If all observations have posterior probability 1, then they are expected to come from the positive class.

    If all observations have posterior probability 0, then they are not expected to come from the positive class.

    Tips

    Here is one way to predict positive class posterior probabilities.

    1. Train an SVM classifier by passing the data to fitcsvm. The result is a trained SVM classifier, such as, SVMModel, that stores the data. The software sets the score transformation function property (SVMModel.ScoreTransformation) to none.

    2. Pass the trained SVM classifier SVMModel to fitSVMPosterior or fitPosterior. The result, for example, ScoreSVMModel, is the same, trained SVM classifier as SVMModel, except the software sets ScoreSVMModel.ScoreTransformation to the optimal score transformation function.

      If you skip step 2, then predict returns the positive class score rather than the positive class posterior probability.

    3. Pass the trained SVM classifier containing the optimal score transformation function (ScoreSVMModel) and predictor data matrix to predict. The second column of the second output argument stores the positive class posterior probabilities corresponding to each row of the predictor data matrix.

    Algorithms

    If you reestimate the score-to-posterior-probability transformation function, that is, if you pass an SVM classifier to fitPosterior or fitSVMPosterior and its ScoreTransform property is not none, then the software:

    • Displays a warning

    • Resets the original transformation function to 'none' before estimating the new one

    References

    [1] Platt, J. "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods". In: Advances in Large Margin Classifiers. Cambridge, MA: The MIT Press, 2000, pp. 61–74.

    Was this topic helpful?