Bioinformatics Toolbox™

classperf - Evaluate performance of classifier

Syntax

classperf
CP = classperf(truelabels)
CP = classperf(truelabels, classout)
CP = classperf(..., 'Positive', PositiveValue, 'Negative', NegativeValue)
classperf(CP, classout)
classperf(CP, classout, testidx)

Arguments

Return Values

Description

classperf provides an interface to keep track of the performance during the validation of classifiers. classperf creates and, optionally, updates a classifier performance object, CP, which accumulates the results of the classifier. The performance properties of a classifier performance object are listed in the following table.

classperf, without input arguments, displays all the performance properties of a classifier performance object.

CP = classperf(truelabels) creates and initializes an empty classifier performance object. CP is the handle to the object. truelabels is a vector or cell array of strings containing the true class labels for every observation. When used in a cross-validation design experiment, truelabels must have the same size as the total number of observations.

CP = classperf(truelabels, classout) creates CP using truelabels, then updates CP using the classifier output, classout.

Tip   This syntax is useful when you want to know the performance of a single validation.

CP = classperf(..., 'Positive', PositiveValue, 'Negative', NegativeValue) specifies the positive and negative labels to identify the target and the control classes, respectively. These labels are used to compute clinical diagnostic test performance.

If truelabels is a numeric vector, PositiveValue and NegativeValue must be numeric vectors whose entries are subsets of grp2idx(truelabels). If truelabels is a cell array of strings, PositiveValue and NegativeValue can be cell arrays of strings or numeric vectors whose entries are subsets of grp2idx(truelabels). PositiveValue defaults to the first class returned by grp2idx(truelabels), while NegativeValue defaults to all other classes.

PositiveValue and NegativeValue must consist of disjoint sets of the labels used in truelabels. For example, if

truelabels = [1 2 2 1 3 4 4 1 3 3 3 2]

you could set

p = [1 2];
n = [3 4];

For example, if you have a data set with data from six samples: five different types of cancer (ovarian, lung, prostate, skin, brain) and no cancer, then ClassLabels = {'Ovarian', 'Lung', 'Prostate', 'Skin', 'Brain', 'Healthy'}.

You could test a detector for lung cancer by using a PositiveValue of 2, and a NegativeValue = [1 3 4 5 6].

Or you can test for any type of cancer by using PositiveValue = [1 2 3 4 5] and a NegativeValue of 6.

In clinical tests, inconclusive values such as '' or NaN are counted as false negatives for the computation of the specificity, and as false positives for the computation of the sensitivity. That is, inconclusive results may decrease the diagnostic value of the test. Tested observations for which truelabels is not within the union of PositiveValue and NegativeValue are not considered. However, tested observations that result in a class not covered by the vector truelabels are counted as inconclusive.

classperf(CP, classout) updates CP, the classifier performance object, with the classifier output classout. classout must be the same size as truelabels, the vector or cell array used to construct the classifier performance object. When classout is a cell array of strings, an empty string, '', represents an inconclusive result of the classifier. For numeric arrays, NaN represents an inconclusive result.

classperf(CP, classout, testidx) updates CP, the classifier performance object, with the classifier output classout. classout has a smaller size than truelabels. testidx is an index vector or a logical index vector of the same size as truelabels, the vector or cell array used to construct the classifier performance object. testidx indicates the observations that were used in the current validation.

Note   In the two previous syntaxes, you do not need to create a separate output variable to update the classifier performance object, CP.

Properties of a Classifier Performance Object

You can access classifier performance object properties by using the get function

get(CP, 'ControlClasses')

or using dot notation

CP.ControlClasses

You cannot directly modify the classifier performance object properties by using the set function, with the exception of the Label and Description properties.

Tip   To modify properties, use either of the following syntaxes: classperf(CP, classout) classperf(CP, classout, testidx)

Example Diagnostic Table

In a cancer study of ten patients, suppose we get the following results:

The diagnostic table would look as follows:

Examples

% Classify the fisheriris data with a K-Nearest Neighbor classifier
load fisheriris
c = knnclassify(meas,meas,species,4,'euclidean','Consensus');
cp = classperf(species,c)
get(cp)
 
% 10-fold cross-validation on the fisheriris data using linear
% discriminant analysis and the third column as only feature for
% classification
load fisheriris
indices = crossvalind('Kfold',species,10);
cp = classperf(species); % initializes the CP object
for i = 1:10
    test = (indices == i); train = ~test;
    class = classify(meas(test,3),meas(train,3),species(train));
    % updates the CP object with the current classification results
    classperf(cp,class,test)  
end
cp.CorrectRate % queries for the correct classification rate

 
cp =
 
	biolearning.classperformance

                        Label: ''
                  Description: ''
                  ClassLabels: {3x1 cell}
                  truelabels: [150x1 double]
         NumberOfObservations: 150
               ControlClasses: [2x1 double]
                TargetClasses: 1
            ValidationCounter: 1
           SampleDistribution: [150x1 double]
            ErrorDistribution: [150x1 double]
    SampleDistributionByClass: [3x1 double]
     ErrorDistributionByClass: [3x1 double]
               CountingMatrix: [4x3 double]
                  CorrectRate: 1
                    ErrorRate: 0
             InconclusiveRate: 0.0733
               ClassifiedRate: 0.9267
                  Sensitivity: 1
                  Specificity: 0.8900
      PositivePredictiveValue: 0.8197
      NegativePredictiveValue: 1
           PositiveLikelihood: 9.0909
           NegativeLikelihood: 0
                   Prevalence: 0.3333
              DiagnosticTable: [2x2 double]


ans =
    0.9467

See Also

Bioinformatics Toolbox functions: crossvalind, knnclassify, svmclassify

Statistics Toolbox™ functions: classify, grp2idx

chromosomeplot

cleave