4 views (last 30 days)

I'm trying to create an RCNN that compares two image samples (each 40x40x3, in the example below) and then gives a numerical response.

I feel that the two halves of the model need to be segregated until quite a way along and then joined using a cat layer.

The problem I have is that I can't seem to find a way to make the model load two images to two seperate sides of the model.

Mahmoud Afifi's code was a useful introduction to my problem: https://uk.mathworks.com/matlabcentral/answers/369328-how-to-use-multiple-input-layers-in-dag-net-as-shown-in-the-figure#answer_398533

However, this relies upon an input format that doesn't seem to work with a regression output layer, it uses an imageDataStore, which in turn specifies a custom image loader "inRead" < imds = imageDatastore(trainImages.imageFilename, 'LabelSource', 'foldernames','IncludeSubfolders',true,'ReadFcn',@inRead); >

I've tried adapting Mahmoud's code to take a single image and cut it in half and then output the left and right half of the image to the two sides of the RCNN. This seems to work in terms of functionality. But I get an error from the "analyse" button on the Deep Network Designer. I also get an error when I try to train it, see below.

I get the following error:

Error using trainNetwork (line 170)

Invalid network.

Caused by:

Layer 'leftConv': Input size mismatch. Size of input to this layer is different from the expected input size.

Inputs to this layer:

from layer 'leftSplitter' (40×40 output)

Layer 'rightConv': Input size mismatch. Size of input to this layer is different from the expected input size.

Inputs to this layer:

from layer 'rightSplitter' (40×40 output)

As the input layer image is 40x80x3 and the subsequent output of the splitter items is 40x40* matlab seems to 'know' the size of the convolution layer should be, yet it doesn't work, anyone have any suggestions?

There's some demo code below, that creates a smalll model below which should learn to return the difference in mean value between the left and right image. No idea if the architecture is correct for this kind of problem, I just knocked it together to show you the error I'm getting.

Thanks,

George

* not sure what happened to the colour planes? Should it be 40x40x3 is that the problem?

% Some simple code to demonstrate the problem

imageSize = [40 80 3]

input = imageInputLayer(imageSize,'Name','InputLayer','Normalization','zerocenter');

layerL=splittingLayerLeftRight('leftSplitter','left')

layerR=splittingLayerLeftRight('rightSplitter','right')

RtwodConvLayer = convolution2dLayer(5, 32, 'Padding', 2, ...

'BiasLearnRateFactor', 2, 'Weights', ones([5 5 1 32]),'Name','rightConv','numChannels',1);

LtwodConvLayer = convolution2dLayer(5, 32, 'Padding', 2, ...

'BiasLearnRateFactor', 2, 'Weights', ones([5 5 1 32]),'Name','leftConv','numChannels',1);

leftCol = [input; layerL;LtwodConvLayer]

rightCol= [layerR;RtwodConvLayer]

% outputlayers

% cat convs

numInputs = 2;

cat_dim = 3; %third dimension

cat_Layer = concatenationLayer(cat_dim,numInputs,'Name','Cat-Layer');

fc1 = fullyConnectedLayer(1024,'Name', 'FC-1');

relu1 = reluLayer('Name','ReLu-FC-1');

dropout1 = dropoutLayer('Name','dropOut-FC-1');

fcWidth=10;

fc = fullyConnectedLayer(fcWidth,'Name', 'FC-out');

%softmxLayer = softmaxLayer('Name','Softmaxx');

%endLayer = classificationLayer('Name','outLayer');

rLayer = regressionLayer("Name","regressionoutput");

outputLayers = [ cat_Layer

fc1

relu1

dropout1

fc

rLayer];

%fullModel = [leftCol; outputLayers; rightCol]

layers= layerGraph([leftCol; outputLayers]);

layers= addLayers(layers,rightCol);

% % layers = connectLayers(layers,sprintf('rightConv',...

% % length(layerDepths),length(layerDepths)),'Cat-Layer/in2');

layers = connectLayers(layers,'rightConv','Cat-Layer/in2');

% connect input to column 2

layers = connectLayers(layers,'InputLayer','rightSplitter');

tempoutputfolder = tempname;

mkdir(tempoutputfolder)

numImages = 100;

filenames = {};

imDiffs = [];

for i=1:numImages

leftImg = rand(40,40,3)+rand;

rightImg = rand(40,40,3)+rand;

filenames{i} = [tempoutputfolder filesep 'inputImg_' num2str(i) '.png'];

imwrite([leftImg rightImg],filenames{i});

imDiffs(i) = mean(leftImg(:))-mean(rightImg(:));

end

trainData = table(filenames',imDiffs');

MiniBatchSize= 128;

InitialLearnRate= 1e-3;

LearnRateSchedule= 'piecewise';

LearnRateDropFactor= 0.1;

MaxEpochs = 3000;

LearnRateDropPeriod= 1000;

Verbose= true;

ValidationFrequency=100;

Verbose=false;

Plots='training-progress';

options = trainingOptions('sgdm', ...

'MiniBatchSize', MiniBatchSize, ...

'InitialLearnRate', InitialLearnRate, ...

'LearnRateSchedule', LearnRateSchedule, ...

'LearnRateDropFactor', LearnRateDropFactor, ...

'LearnRateDropPeriod', LearnRateDropPeriod, ...

'MaxEpochs', MaxEpochs, ...

'Verbose', Verbose, ...

'Verbose',Verbose,...

'Plots',Plots);

rcnn=trainNetwork(trainData,layers,options)

% Modified by George Lovell from original code written by Mahmoud Afifi -- mafifi@eecs.yorku.ca | m.3afifi@gmail.com

% Split an image into left/right halves for output to a network with

% multiple image inputs.

%

% Requires Matlab 2019b or higher

classdef splittingLayerLeftRight < nnet.layer.Layer

properties

target

end

properties (Learnable)

end

methods

function layer = splittingLayerLeftRight(name,target)

layer.Name = name;

layer.Description = "splittingLayerLeftRight";

layer.target = target;

end

function Z = predict(layer, X)

imWidth = size(X,2);

if rem(imWidth,2)~=0

error('To split an image into two left/right halves it needs to have an even width');

else

imHalf=imWidth/2;

end

switch layer.target

case 'left'

Z = X(:,1:imHalf,1:3);

case 'right'

Z = X(:,imHalf+1:end,1:3);

end

%figure;imagesc(Z);

end

end

end

Sign in to answer this question.

Opportunities for recent engineering grads.

Apply Today
## 2 Comments

## Direct link to this comment

https://www.mathworks.com/matlabcentral/answers/495982-creating-an-rcnn-with-two-image-inputs-and-a-regression-output#comment_777294

⋮## Direct link to this comment

https://www.mathworks.com/matlabcentral/answers/495982-creating-an-rcnn-with-two-image-inputs-and-a-regression-output#comment_777294

## Direct link to this comment

https://www.mathworks.com/matlabcentral/answers/495982-creating-an-rcnn-with-two-image-inputs-and-a-regression-output#comment_777352

⋮## Direct link to this comment

https://www.mathworks.com/matlabcentral/answers/495982-creating-an-rcnn-with-two-image-inputs-and-a-regression-output#comment_777352

Sign in to comment.