Which kind of Deep Learning architecture (CNN, LSTM) could I use for classification duty of monodimension signal?

Hello, I am trying to classify monodimensional signals (spectrum information) using Deep Learning algorithm. Having a dataset of 12000 observation, of 1x2048 samples (frequency taps), I tried to use CNN (NN toolbox of Matlab), with different convolution layer, without good result. I even tried to use LSTM but nothing change. Any suggestion?

Thanks in advance.

Answer 1

[Vishal Bhutani]

By my understanding, you want to train a Neural Network to classify one-dimensional signals. One of the thing you can try is Deep Neural Network with multiple hidden layers, there are various hyperparameter which you can vary: learning rate, number of neurons, number of hidden layers and if you are using recent MATLAB version you can vary the optimizer also same for LSTM. For CNN, try varying the size of filters, number of filters and learning rate. For 1-D data, mostly DNN or LSTM work, but you can try various networks. If possible try increasing dataset.

[Alessio Izzo]

Hi Mr Vishal, yes I tried already to change some of this parameters. Since easy architeture did not work well I started to get a deep architecture, like this: layers = [ ... imageInputLayer([1 2048 2]) convolution2dLayer([1 5],2) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],4) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],16) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],32) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],64) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],128) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 3],256) reluLayer maxPooling2dLayer([1 2],'Stride',2) convolution2dLayer([1 1],512) reluLayer fullyConnectedLayer(2) softmaxLayer classificationLayer]; options = trainingOptions('sgdm',... 'Momentum',0.95,... 'InitialLearnRate',0.1,... 'LearnRateSchedule','piecewise',... 'LearnRateDropFactor',0.01,... 'LearnRateDropPeriod',5,... 'L2Regularization',0.0001,... 'MaxEpochs',65,... 'MiniBatchSize',100,... 'Plots','training-progress');

But, still the accuracy was around the 50% (the worst result in my opinion). So after, a short reading on sequence classification I tried to use the LSTM, with an architecture as report below: inputSize = 2; numHiddenUnits = 100; numClasses = 2;

layers = [ ... sequenceInputLayer(inputSize) bilstmLayer(numHiddenUnits,'OutputMode','sequence') fullyConnectedLayer(numClasses) softmaxLayer classificationLayer];

maxEpochs = 100; miniBatchSize = 100;

options = trainingOptions('adam', ... 'ExecutionEnvironment','gpu', ... 'InitialLearnRate',0.8,... 'GradientThreshold',1, ... 'MaxEpochs',maxEpochs, ... 'MiniBatchSize',miniBatchSize, ... 'SequenceLength','longest', ... 'Plots','training-progress'); But nothing change yet. Do you belive that 12000 are not enough observations? Even if there are only 2 classes to be classified? Any other suggestion?

Kind Regards, Alessio

[Vishal Bhutani]

Since there is no significant changes are observed in accuracy while changing architecture, one thing you can try is increasing the dataset if possible. If the accuracy remains same then try for more complex models by combining various networks and then classify later. Another thing you can try is to calculate some features from your data and then train the model. And also you can try Deep Neural Network with multiple hidden layers. Hope it helps.

[Alessio Izzo]

Hi I am trying to implemente this architecture now:

inputSize = 1;

numHiddenUnits = 100;

numClasses = 2;

layers = [ ...

    sequenceInputLayer(inputSize)
    bilstmLayer(numHiddenUnits,'OutputMode','sequence')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

maxEpochs = 100;

miniBatchSize = 100;

options = trainingOptions('adam', ...

    'ExecutionEnvironment','gpu', ...
    'InitialLearnRate',0.8,...
    'GradientThreshold',1, ...
    'MaxEpochs',maxEpochs, ...
    'MiniBatchSize',miniBatchSize, ...
    'SequenceLength','longest', ...
    'Plots','training-progress');

net = trainNetwork(train_data_cell,categorical_label_new,layers,options);

But at the end I have an error saying:

Invalid training data. If all recurrent layers have output mode 'sequence', then the responses must be a cell array of categorical sequences, or a categorical sequence.

Now my training data is a cell 12000x1 with each observation long 1x2048. The labels are in a categorical array with value 0 and 1 of dimension 12000x1. If I change the output mode from 'sequence' to 'last' it works. What is the problem?

[Vishal Bhutani]

Hi, based on my understanding for sequence-to-label classification, it is mention in the documentation that use the ‘OutputMode’ to be ‘last’. For sequence-to-sequence regression, use ‘OutputMode’ to be ‘sequence’. Here is the documentation link for that :

https://www.mathworks.com/help/nnet/ug/long-short-term-memory-networks.html#mw_9cf7b3dc-bc5e-40f2-a20a-82bdd6df1b03

Hope it helps.

[Alessio Izzo]

Yeah you are right, I saw it yesterday and It works.

Thanks for your support Vishal.

[Alessio Izzo]

What about the validation of the LSTM? I saw it is not possibile to have the 'ValidationData' into the trainingOption. An alternative is to use CheckPoints and OutputFcn to (once per epoch) load the network from a checkpoint and run it against my validation data. But I did not manage to make it working. Any idea?