CNN seems to be too inaccurate to classify my images. Is there an alternative that I can explore?

Hi folks,

I've attached examples of 2 different classifications for images that I'm working with below. My CNN (and a retrained Alexnet) seem to be too inaccurate to classify the images reliably.

Is there an alternative image classification technique that you'd recommend I use?

Thanks in advance

Classification 1

Classification 2

Answer 1

[Daniel Vieira]

Alexnet is the simplest of the pretrained deep learning models, you might want to try others (resnet, inception, etc).

You might also not have enough images to get a good model, I don't know how many you have.

You might be experiencing overfitting, how went the training? A near 100% accuracy in training data with not that much in validation data would be a pretty strong indication of overfitting. You can avoid overfitting with image augmentation, dropout layers, etc.

Also, some problems have inherently ambiguous classes. A network to classify musical genders in song files might have a hard time in separating similar musical subgenders, for example. You might take some time to think if this is not your case. If it is, your network might not be "innacurate" at all, you problem is inherently difficult instead. The prediction class scores might give an idea of such ambiguities: 2 classes appearing with average-ish scores, indicating the image might belong to either.

[Teshan Rezel]

Hi there,

I have 1000 images per class for training and validation, and it reaches 99.6% over 20 epochs with a learn rate of 0.001.

I also tried making my own CNN but the results are the same:

AnisotropyDatasetPath = fullfile(matlabroot,'Training', 'Anisotropy');
IsotropyDatasetPath = fullfile(matlabroot,'Training', 'Isotropy');
FillerDatasetPath = fullfile(matlabroot,'Training', 'Filler');
TrainingDatasetPath = fullfile(matlabroot,'Training');

imds = imageDatastore(TrainingDatasetPath, 'IncludeSubfolders',true,...
    'LabelSource','foldernames');


labelCount = countEachLabel(imds)

numTrainFiles = 500;
[imdsTrain,imdsValidation] = splitEachLabel(imds,numTrainFiles,'randomize');

layers = [
    imageInputLayer([227 227 3])
    
    convolution2dLayer(3, 8, 'Padding', 'same')
    batchNormalizationLayer
    reluLayer
    
    maxPooling2dLayer(2, 'Stride', 2)
    
    convolution2dLayer(3, 16, 'Padding', 'same')
    batchNormalizationLayer
    reluLayer
    
    maxPooling2dLayer(2, 'Stride', 2)
    
    convolution2dLayer(3, 32, 'Padding', 'same')
    batchNormalizationLayer
    reluLayer
    
    fullyConnectedLayer(3)
    softmaxLayer
    classificationLayer];

options = trainingOptions('sgdm', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs', 20, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'Plots','training-progress');

net = trainNetwork(imdsTrain, layers, options);

YPred = classify(net, imdsValidation);
YValidation = imdsValidation.Labels;

accuracy = sum(YPred == YValidation)/numel(YValidation)

[Daniel Vieira]

99.6% seems rather accurate to me, does your application really need more...? Or is this only on the training data, and it's performing poorly on test data?

also, I noticed that, by your MaxEpochs, MiniBachSize and ValidationFrequency parameters, you only let your network make 2 validations, it is probably finishing training by number of Epochs instead of validation criteria, which is a bit pointless. I would let it make validations more often, like every 5 or 10 iterations, and increase MaxEpochs. Then also set a ValidationPatience value to 5 or 10 (default is infinite, which doesn't help at all).

Another thing that might help is setting learn rate decay parameters (LearnRateSchedule, LearnRateDropFactor and LearnRateDropPeriod). Setting these parameters makes learning slow down as it goes on, effectively "refining" your model as it nears the end of training.

you could also try other training optimizers, sgdm is only 1 of them. I feel adam tends to do a better job (but I admit this is just a feeling). Adam optimizer already has a built-in learn rate "modulation" feature, so it doesn't use the LearnRateSchedule, LearnRateDropFactor and LearnRateDropPeriod parameters.