KFoldLoss: Get separate Loss for Classes

Hi,

I'm trying to compare some models using k-fold-cross-validation.

My problem is that I'd like to get the loss for my binary classes separatly (a TPR per class), as I have quite a big class-imbalance and the overall loss isn't representative for the model performance.

% Lbl and FV need to be initialized prior

Partitions = cvpartition(Lbl,'KFold',10); %Partition Data into 10 Folds

indResClass1 = zeros(10,6); %initialize Matrix of individual Fold results for Class1
%indResClass2 = zeros(10,6); %initialize Matrix of individual Fold resultsfor Class2

%Train and validate the example Models
FineKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',1,'Standardize',1);
indResClass1(1:10,1) = ones(10,1) - kfoldLoss(FineKNN,'Mode','individual');

MediumKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',10,'Standardize',1);
indResClass1(1:10,2) = ones(10,1) - kfoldLoss(MediumKNN,'Mode','individual');

CoarseKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',100,'Standardize',1);
indResClass1(1:10,3) = ones(10,1) - kfoldLoss(CoarseKNN,'Mode','individual');

CosineKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',10,'Distance','cosine','Standardize',1);
indResClass1(1:10,4) = ones(10,1) - kfoldLoss(CosineKNN,'Mode','individual');

CubicKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',10,'Distance','minkowski','Standardize',1);
indResClass1(1:10,5) = ones(10,1) - kfoldLoss(CubicKNN,'Mode','individual');

WeightedKNN = fitcknn(FV,Lbl,'CVPartition',Partitions,'NumNeighbors',10,'DistanceWeight','squaredinverse','Standardize',1);
indResClass1(1:10,6) = ones(10,1) - kfoldLoss(WeightedKNN,'Mode','individual');

%Output Boxplot
boxplot(indResClass1,'Labels',{'Fine KNN','Medium KNN','Coarse KNN','Cosine KNN',...
    'Cubic KNN','Weighted KNN'})
xlabel('Trained Models')
ylabel('Cross-Validation TPR')

My code for now partitions the data, trains some partitioned example models and validates them for a boxplot output. Problem is still that I'd like to get the separate losses in two different plots (see indResClass2 which is uncommented by now).

Thanks in advance for any kind of help on this! :)