how to implement subpixel-conv2D?

Is there any command available in MATLAB to implement subpixel-conv2D layer, as proposed in

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, Zehan Wang, " Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" , arXiv:1609.05158 [cs.CV].

Kindly direct me to its link, if it is available, or give any suggestions for its implementation in the decoder side of U-Net.

Answer 1

[Nithin]

Hi @Sania Gul,

MATLAB does not have a built-in "subpixelConv2dLayer" as a ready-to-use layer in the "Deep Learning Toolbox". The only workaround is to manually implement a Subpixel Convolution layer as a custom layer, often called a "PixelShuffle" operation. This is exactly what "Wenzhe Shi et al." proposed in their paper, they use sub-pixel convolution to efficiently upscale images by reshaping feature maps rather than using deconvolution or interpolation.

Refer to the following code to understand how to define a Custom Subpixel Layer:

classdef SubpixelLayer < nnet.layer.Layer 
    properties 
        Scale 
    end 
    
    methods 
        function layer = SubpixelLayer(scale, name) 
            layer.Name = name; 
            layer.Description = "Subpixel layer with scale " + scale; 
            layer.Scale = scale; 
        end 
        
        function Z = predict(layer, X) 
            % X is size HxWx(C*r^2)xN 
            r = layer.Scale; 
            [H, W, C_mul_r2, N] = size(X); 
            C = C_mul_r2 / (r^2); 
            
            if mod(C_mul_r2, r^2) ~= 0 
                error('Number of channels must be divisible by scale^2'); 
            end 
            
            X = reshape(X, H, W, r, r, C, N); 
            X = permute(X, [1 3 2 4 5 6]);  % H, r, W, r, C, N 
            X = reshape(X, H*r, W*r, C, N); 
            Z = X; 
        end 
    end 
end 

This "SubpixelLayer" can be integrated as a new layer in the model, provided that the decoder includes a convolutional layer producing "C × r²" output channels:

% r is the upscale factor 
convLayer = convolution2dLayer(3, numChannels * r^2, 'Padding', 'same'); 
subpixelLayer = SubpixelLayer(r, 'subpixel'); 

% Add to layerGraph 
lgraph = addLayers(lgraph, [ 
    convLayer 
    subpixelLayer 
    ]); 

Kindly refer to the following MATLAB documentation to understand more about defining custom Deep Learning Layers with Learnable Parameters: https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html

[Sania Gul]

Tnx a lot. U solved the mystery, not only for this problem but also for other deep python networks' layers implementation in Matlab :-).