How can I create an ROC plot from the set of cross-validation models using Statistics Toolbox 7.5 (R2011a)?

I am using the CROSSVAL command to validate the performance of 'NaiveBayes' model for my data. I am computing the metrics, misclassification rate and confusion matrix using stratified 10-fold cross-validation to analyze the performance of the classifier. I intend to create an ROC curve from the validation results. Is there any way to create ROC plot from the cross-validation results of my model using Statistics Toolbox 7.5 (R2011a)?

Answer 1

[MathWorks Support Team]

In order to plot ROC curve, the decision boundary should be a value of the posterior probabilities that you compute using ‘NaiveBayes’. You can plot ROC curves using PERFCURVE, which computes the ROC curve for a vector of classifier predictions, given the true class. More details about PERFCURVE are available via the following link:

https://www.mathworks.com/help/stats/perfcurve.html

Here is a small example of how to plot an ROC curve without cross-validation:

     load fisheriris
     x = meas(51:end,1:2);
     y = species(51:end);
     nb = NaiveBayes.fit(x,y);
     p = posterior(nb,x);
     [X,Y] = perfcurve(y,p(:,1),'versicolor');
     plot(X,Y)
     xlabel('False positive rate'); ylabel('True positive rate')
     title('ROC for classification by naïve Bayes')

To use 10-fold cross-validation, you can fit the model on 90% of the data, and compute predicted posteriors for the remaining 10% of data which was not used for fitting. You can then loop over each of the 10 subsets to plot the ROC curves for individual runs.