265 views (last 30 days)

Show older comments

I am trying to classify image datasets using deep learning.

after getting feature vector of each single image I ve got a matrix 18000x24000 which indicates to No. of images x features.

I used:

trainNetwork (X, Y, Layers, Options)

Where X is the train data and Y is the Labels which is 18000x1. But there is an error says Invalid training data. X and Y must have the same number of observations.

I think I should change the train matrix to 4-D but I don't know how, and if it is correct?

Ali Yar Khan
on 6 Dec 2019

i have row of feature in data and one distinct value in label for each row of data ... my network is also giving me the same error ... here is my code.

%loading train and test data

trainingData = load('train.mat');

trainData = trainingData.data;

trainLabels = categorical(trainingData.labels);

testingData = load('test.mat');

testData = testingData.data;

testLabels = testingData.labels;

% Define Network Architecture

% Define the convolutional neural network architecture.

layers = [

imageInputLayer([1 144 1]) % 22X1X1 refers to number of features per sample

convolution2dLayer(3,16,'Padding','same')

reluLayer

fullyConnectedLayer(384) % 384 refers to number of neurons in next FC hidden layer

fullyConnectedLayer(384) % 384 refers to number of neurons in next FC hidden layer

fullyConnectedLayer(2) % 2 refers to number of neurons in next output layer (number of output classes)

softmaxLayer

classificationLayer];

options = trainingOptions('sgdm',...

'MaxEpochs',500, ...

'Verbose',false,...

'Plots','training-progress');

%train the network

net = trainNetwork(trainData,trainLabels,layers,options);

%test the network

predictedLabels = classify(net,trainD)'

accuracy = sum(predictedLabels == testLabels)/numel(testLabels)

It gives me this error

Invalid training data. X and Y must have the same number of observations.

Error in convnet1d (line 27)

net = trainNetwork(trainData,trainLabels.',layers,options);

Caused by:

Error using nnet.internal.cnn.util.NetworkDataValidator/assertXAndYHaveSameNumberOfObservations (line 142)

Invalid training data. X and Y must have the same number of observations.

Any help regarding this . I have this work space ... that may help you about the dimensions of data

Bhartendu
on 8 Apr 2018

Try the following:

If your data ( 18000 data points) is in form of images with dimensions say, 120*200 (equals to 24000), then reshape it as follows:

X_train = reshape(X, [120,200,1,size(X,1)]);

This should create 4-D Matrix X_train of size (120,200,1,18000), then train the network using:

net = trainNetwork(X_train,Y,Layers,Options)

Radians
on 5 Feb 2020

I have done exactly that but to no avail.

I

where 'X_1' is my X and 'correspondingvalues' is my Y. basically, x is a 16*9500 items collection of 16*16 images. Y is their labels.

Please help.

US
on 26 Jul 2020

Edited: US
on 26 Jul 2020

I am also getting the same error (my layers structure is a prediction CNN):

>> whos X Y

Name Size Bytes Class Attributes

X 64x64x52x1800 3067084800 double

Y 1800x1 14400 double

>> MyPrediction = trainNetwork(X, Y, layers, opts);

Error using trainNetwork (line 165)

Number of observations in X and Y disagree.

It gives me the same error when I transpose Y, or when I reshape X to [64*64,52,1,1800].

And I have:

MATLAB Version: 9.6.0.1072779 (R2019a)

I am starting to think that this MATLAB function trainNetwork or a sub-function that it calls is somehow not behaving properly or I am missing something.

Any help will be appreciated.

michael scheinfeild
on 9 Jun 2018

hi i have similar issue and cant solve it i also look at the example Sequence Classification Using Deep Learning

i try to classify between different signals based frequency future

%%generete the signal

fs=500;

f=20;

t=[0:1/fs:5-1/fs];

xsig=10*sin(2*pi*f*t);

sampleLen=length(xsig);

NFFT = 2.^nextpow2(sampleLen);

% signal fft

hwin=hamming(length(xsig))';

Y = fft(xsig.*hwin,NFFT);

magnitudeY = abs(Y);

xsampleFFT=20*log10(magnitudeY(1:NFFT/2));

figure,plot(xsampleFFT);title('signal fft')

% noise fft

xnoise= randn(size(xsig));

Yn = fft(xnoise.*hwin,NFFT);

magnitudeYn = abs(Yn);

xnoiseFFT=20*log10(magnitudeYn(1:NFFT/2));

figure,plot(xnoiseFFT);title('noise')

%%make the data

sampleLen = 100;

NFFT = 128;

hwin=hamming(sampleLen)';

nsamples=length(xsig)/sampleLen;

xsampleFFT={};%zeros(nsamples,NFFT/2);

hwin=hamming(sampleLen)';

kj=1;

for(k=1:sampleLen:length(xsig)-sampleLen+1)

cursig=xsig(k:k+sampleLen-1);

Y = fft(cursig.*hwin,NFFT);

magnitudeY = abs(Y); % Magnitude of the FFT

xTrain{kj}=20*log10(magnitudeY(1:NFFT/2));

%figure,plot(xTrain{kj})

yTrain(kj)=categorical(1);

kj=kj+1;

end

disp('data types')

[size(xTrain) size(yTrain)]

[size(xTrain{1})]

class(xTrain)

class(yTrain)

%append

for(k=1:sampleLen:length(xnoise)-sampleLen+1)

cursig=xnoise(k:k+sampleLen-1);

Y = fft(cursig.*hwin,NFFT);

magnitudeY = abs(Y); % Magnitude of the FFT

xTrain{kj}=20*log10(magnitudeY(1:NFFT/2));

%figure,plot(xTrain{kj})

yTrain(kj)=categorical(0);

kj=kj+1;

end

disp('data types')

[size(xTrain) size(yTrain)]

[size(xTrain{1})]

class(xTrain)

class(yTrain)

figure,plot(yTrain)

figure,subplot(2,1,1),plot(xTrain{10})

subplot(2,1,2);plot(xTrain{30})

function [net] = train_lstm(XTrainLoc,YTrainLoc)

%%lstm

inputSize = 1;

numHiddenUnits = 100;

numClasses = 2;

layers = [ ...

sequenceInputLayer(inputSize)

lstmLayer(numHiddenUnits,'OutputMode','sequence')

fullyConnectedLayer(numClasses)

softmaxLayer

classificationLayer];

maxEpochs = 1;

miniBatchSize = 100;

options = trainingOptions('adam', ...

'ExecutionEnvironment','cpu', ...

'GradientThreshold',1, ...

'MaxEpochs',maxEpochs, ...

'MiniBatchSize',miniBatchSize, ...

'SequenceLength','longest', ...

'Shuffle','never', ...

'Verbose',0, ...

'Plots','training-progress');

% options = trainingOptions('adam', ...

% 'ExecutionEnvironment','cpu', ...

% 'GradientThreshold',1, ...

% 'MaxEpochs',maxEpochs, ...

% 'MiniBatchSize',miniBatchSize, ...

% 'SequenceLength','longest', ...

% 'Shuffle','never', ...

% 'Verbose',0, ...

% 'Plots','training-progress',...

% 'ValidationData',{XValidation,YValidation},...

% 'ValidationPatience',Inf);

%%train

net = trainNetwork(XTrainLoc,YTrainLoc,layers,options);

====== i recive error

[net] = train_lstm(xTrain,yTrain)

*Error using trainNetwork (line 154)

Invalid training data. If all recurrent layers have output

mode 'sequence', then the responses must be a cell array of

categorical sequences, or a categorical sequence.*

_Error in train_lstm (line 42) net = trainNetwork(XTrainLoc,YTrainLoc,layers,options);

Caused by: Error using nnet.internal.cnn.util.NetworkDataValidator/assertOutputModeCorrespondsToDataForClassification (line 380) Invalid training data. If all recurrent layers have output mode 'sequence', then the responses must be a cell array of categorical sequences, or a categorical sequence._

so what is the issue i tried also change y to cell array of category , transpose the internal x, change network in. i think in this fft i have actually one sample each time with nfft feature. this is the same as the Japanese sample but they have 12 features

SARAH LONER
on 29 Nov 2019

sir i also have the same issue cant able to train network for an image .

my work was image segmentation based on unet segment cant able to clear the code

it showing error at

net = trainNetwork(imds,layers,options);

where in imds i have taken image

kindly help to solve this error

SARAH LONER
on 30 Nov 2019

inp=b % my input image is dicom image

DatasetPath=fullfile('C:\Users\Desktop\to');

imds=imageDatastore(DatasetPath, 'IncludeSubfolders', true,...

'LabelSource','foldernames','fileextension',{'.dcm'});

labelDir = fullfile(DatasetPath,'testImages');

I = readimage(imds,1);

I = histeq(I);

imshow(I)

classes = [

"MALIGNANT","BENIGN"

];

labelIDs=[255 0]

inputlayer = imageInputLayer([512 512 1],'Name','inp')

numFilters = 64;

numLayers = 16;

layers = [ ...

imageInputLayer([512 512 1])

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

transposedConv2dLayer(4,numFilters,'Stride',2,'Cropping',1);

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

transposedConv2dLayer(4,numFilters,'Stride',2,'Cropping',1);

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

convolution2dLayer(5,20)

fullyConnectedLayer(4)

softmaxLayer

pixelClassificationLayer

]

pxds = pixelLabelDatastore(labelDir,classes,labelIDs);

options = trainingOptions('sgdm', ...

'InitialLearnRate',0.01, ...

'MaxEpochs',1, ...

'Shuffle','every-epoch', ...

'ValidationFrequency',30, ...

'Verbose',false);

ds = pixelImageDatastore(imds,pxds);

net=trainNetwork(ds,layers,options);

I = read(imds);

C = read(pxds)

C = semanticseg(I, net);

% Overlay pixel label data on the image and display.

B = labeloverlay(I, C);

figure(12)

imshow(B)

i got error at

ds = pixelImageDatastore(imds,pxds);

once i cleared that error means again i got error at

net=trainNetwork(ds,layers,options);

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!