error compiling ResNet50 Deep Learning HDL Toolbox Support Package

Hello,

I have a pretrained ResNet50 imported to a DAGNetwork object. The NN is working properly in matlab. However, I'm trying to implement it in a FPGA and I get the followong error while compiling. NN was imported using importTensorFlowNetwork and I'm trying to generate HDL with Deep Learning HDL Toolbox Support Package For Intel FPGA And SoC Devices (I was able to implement a sequential NN using a SeriesNetwork object and this tool):

Error in dnnfpga.apis.Workflow/compile

Error in generate_HDL_from_NN (line 18)
    hW.compile;

Caused by:
    Layer 'conv2_block1_3_conv': Input size mismatch. Size of input to this layer is different from the expected input size.
    Inputs to this layer:
        from layer 'conv2_block1_0_conv' (size 8(S) × 8(S) × 256(C) × 1(B))
    Layer 'conv2_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv2_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv2_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block4_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block4_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block5_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block6_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.

Do you have any suggestion to solve this issue?

Thank you in advance

---------------------------

Software version:

Deep Learning HDL Toolbox Support Package For Intel FPGA And SoC Devices 21.2.4

Matlab R2021b Update 6 (9.11.0.2207237)

Ubuntu 20.04

---------------------------

Answer 1

[Wang Chen]

Hi Ruben,

For Resnet50 network, have you tried to download the Deep Learning Toolbox Model for ResNet-50 Network support pacakge?The Resnet50 network in this support package works together with Deep Learning HDL Toolbox:

https://www.mathworks.com/matlabcentral/fileexchange/64626-deep-learning-toolbox-model-for-resnet-50-network

There is also a ResNet-18 network support pacakge:

https://www.mathworks.com/matlabcentral/fileexchange/68261-deep-learning-toolbox-model-for-resnet-18-network

For the network you imported using importTensorFlowNetwork, could you contact MathWorks customer support and shared the network with customer support? It looks like the network is not correctly constructed in MATLAB. The error message on the "Unconnected input. Each layer input must be connected to the output of another layer" are generic MATLAB network connection errors. They are not specific to FPGA targeting.

Thanks,

Wang

[Rubén]

Hello @Wang Chen,

Thank you for your help. I have installed Deep Learning Toolbox Model for ResNet-50 Network support pacakge but with the same result.

I'm trying to import to matlab in 2 ways:

1. I have exported NN model from TensorFlow Resnet50 + Flatten layer + Dense layer. Importing with importTensorFlowNetwork function I get this error

Error using importTensorFlowNetwork (line 107)
Unable to import the model as a network because it contains custom layers with operators that are not supported. Use
importTensorFlowLayers to import the model as a layer graph.

Error in import_NN (line 18)
    NN = importTensorFlowNetwork(model,'OutputLayerType','classification');

Error in importTensorFlowNN (line 16)
nn = import_NN(model, classNames);
 

Using importTensorFlowLayers and assembleNetwork functions I get:

Error using assembleNetwork (line 47)
Invalid network.

Error in import_NN (line 16)
    NN = assembleNetwork(layers);

Error in importTensorFlowNN (line 16)
nn = import_NN(model, classNames);

Caused by:
    Layer 'resnet50': Error using 'predict' in layer SavedModel.kResnet50Layer45227. The function threw an error and could not be executed.
        Error using SavedModel.kResnet50Layer45227>Resnet50Layer45227 (line 1750)
        Unsupported TensorFlow operator.

        Error in SavedModel.kResnet50Layer45227/predict (line 347)
                    [temp{1}] = Resnet50Layer45227(inputs, obj.conv1ConvKernel, obj.conv1ConvBias, obj.conv1BnGamma, obj.conv1BnBeta, obj.conv1BnMovingMean, ...

2. Exporting NN model from TensorFlow layer by layer. Resnet50 has been splitted into layers (ResNet50 layers + Flatten + Dense layer). Then is when I get the error from the previous post but error is given by hw.compile.

What is the correct procedure to deal with this kind of NN?

Could you confirm that Deel Learning HDL Support Package is also compatible with non-sequential networks (SeriesNetwork matlab object)?

Could you put me through to the MathWorks customer support to share the NN?

Thank you in advance.

[Sivylla Paraskevopoulou]

Hi Ruben,

I think what Wang meant when he suggested to download the Deep Learning Toolbox Model for ResNet-50 Network support package, is to load ResNet50 from the support package instead of importing it.

If you have the support package installed, you can load the network by entering:

net = resnet50

Then, you can generate HDL for net.

[Rubén]

Hello @Sivylla Paraskevopoulou

Thank you for your clarification.

Using Deep Learning Toolbox Model for ResNet-50 Network support package, HDL is compile successfully.

From this, one of the previous post question is answered: "Could you confirm that Deel Learning HDL Support Package is also compatible with non-sequential networks (SeriesNetwork matlab object)? " --> Yes, non-sequential networks and DAGNetwork are compatible with Deel Learning HDL Support Package

However, I need a pretrained network made up of a ResNet50 and some additional layers. How I can import the model and compile the HDL successfully?

I have compared both networks (RestNet50 from Deep Learning Toolbox Model for ResNet-50 Network support package and the custom network I have imported (ResNet50 + reLu + reLu + softmax) and the nly differences I see are in the input and output layers as expected:

[Sivylla Paraskevopoulou]

The displayed error mentions unconnected linputs. When you import the network from TensorFlow, you can check that all the layers are properly connected by using analyzeNetwork. There should be no errors or warnings showing up in Deep Learning Network Analyzer. Also all the layers of the imported network might not support HDL code generation. See Supported Networks, Layers, Boards, and Tools for all supported layers. The network layers might look the same, but the imported network might include custom layers or other layers that do not support HDL code generation.

It may be easier to add the additional desired layers to the MATLAB built-in ResNet50. MATLAB has excellent tools that make model surgery very easy, such as addLayers and connectLayers.

[Rubén]

analyzeNetwork does not give neither warnings nor errors. In addtion nn imported seems working properly, it classify correctly.

My NN uses ResNet50, that is standard and is supported by HDL toolbox, and keras flatten, reLu and softmax layers, that all of them are also supported.

I don't see the point to train Resnet50 out of Matlab and inclue additional layers in Matlab because my model need the additional layers and the training will be different.

[Wang Chen]

Hi Ruben,

It will be great if you can share your imported network to us, so we can take a look at it and see what is the cause of the issue. Could you contact MathWorks customer support and shared the network with customer support?

Alternatively, as Sivylla suggested, you could add the additional desired layers to the MATLAB built-in ResNet50 or ResNet18, using MATLAB tools like Deep Network Designer:

https://www.mathworks.com/help/deeplearning/gs/get-started-with-deep-network-designer.html

Thanks,

Wang

Answer 2

[Rubén]

The size of the NN is over the 5MB limit. Please, find the "NN.mat" in the following download link: NN.mat

[Wang Chen]

Hi Ruben,

Thanks for sharing the network. After looking at your network, we identified that error you saw is caused by a bug in R2021b version. This bug is fixed in R2022b or later version of Deep Learning HDL Toolbox. It is possible for you to upgrade to a R2022b or R2023a version of MATLAB/Deep Learning HDL Toolbox?

Thanks,

Wang

[Rubén]

Hello Wang, Thank you for your support. What is the roocause? Is there any other network impacted by this bug o an specific layer?

At this moment it is not feasible to upgrade Matlab to R2022b because other tool compatibility.

[Rubén]

Hello @Wang Chen,

I'm having problems with others layers and NN. Is the rootcause clear?

[Wang Chen]

Hi Ruben, the root cause of this bug is the pattern of zero padding layer (nnet.keras.layer.ZeroPadding2dLayer) followed by a group convolution layer, please see following bug report: https://www.mathworks.com/support/bugreports/2661508

This bug is fixed in R2022b.

If you cannot upgrade to R2022b, the work-around is to merge the zero padding into the group convolution layer. These two layers can be merged because the group convolution layer also has the padding options.

Thanks,

Wang

[Rubén]

Hello @Wang Chen,

I have imported the SaveModel as a layerGraph ("layers" in this example) object and I tried to merge both layers in this way:

    %% Merge 0 padding layer into groupedConvolution2dLayer layer
    % Copy conv layer and change Padding size
    conv = layers.Layers(3);
    conv.PaddingSize = [layers.Layers(2).Top layers.Layers(2).Bottom layers.Layers(2).Left layers.Layers(2).Right];
    % Replace conv layer and remove padding layer
    layers = replaceLayer(layers, layers.Layers(3).Name, conv);
    layers = removeLayers(layers, layers.Layers(2).Name);
    % Connect input layer with conv layer
    layers = connectLayers(layers, layers.Layers(1).Name, layers.Layers(2).Name)
    
    NN = assembleNetwork(layers);

after doing this, I have built the DAGNetwork object using assembleNetwork and after analising it with analyzeNetwork there is no error/warning.

However, I get the following error whe I try to compile with hW.compile as before:

    Error using dnnfpga.compiler.codegenfpga
Invalid network.

Error in dnnfpga.apis.Workflow/compileNetwork

Error in dnnfpga.apis.Workflow/compile

Error in generate_HDL_from_NN (line 18)
    hW.compile;

Error in importTensorFlowNN (line 36)
hW = generate_HDL_from_NN(nn, ToolName, ToolPath, Vendor, Platform, ip, Interface, User, Pass);

Caused by:
    Layer 'conv2_block1_3_conv': Input size mismatch. Size of input to this layer is different from the expected input size.
    Inputs to this layer:
        from layer 'conv2_block1_0_conv' (size 8(S) × 8(S) × 256(C) × 1(B))
    Layer 'conv2_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv2_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv2_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv3_block4_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block4_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block5_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv4_block6_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block1_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block2_add': Unconnected input. Each layer input must be connected to the output of another layer.
    Layer 'conv5_block3_add': Unconnected input. Each layer input must be connected to the output of another layer.

Find the nn int this link NN_without_padding.mat

Are both layers merged properly?