If you create a custom deep learning layer, then you can use
the checkLayer
function
to check that the layer is valid. The function checks layers for validity, GPU compatibility,
and correctly defined gradients. To check that a layer is valid, run the following
command:
checkLayer(layer,validInputSize,'ObservationDimension',dim)
layer
is an instance of the layer, validInputSize
is a
vector or cell array specifying the valid input sizes to the layer, and dim
specifies the dimension of the observations in the layer input data. For large input sizes, the gradient checks take longer to run. To speed up the tests, specify a smaller valid input size.Check the validity of the example custom layer preluLayer
.
Define a custom PReLU layer. To create this layer, save the file preluLayer.m
in the current folder.
Create an instance of the layer and check that it is valid using checkLayer
. Set the valid input size to the typical size of a single observation input to the layer. For a single input, the layer expects observations of size hbywbyc, where h, w, and c are the height, width, and number of channels of the previous layer output, respectively.
Specify validInputSize
as the typical size of an input array.
layer = preluLayer(20,'prelu');
validInputSize = [5 5 20];
checkLayer(layer,validInputSize)
Skipping multiobservation tests. To enable tests with multiple observations, specify the 'ObservationDimension' option in checkLayer. For 2D image data, set 'ObservationDimension' to 4. For 3D image data, set 'ObservationDimension' to 5. For sequence data, set 'ObservationDimension' to 2. Skipping GPU tests. No compatible GPU device found. Running nnet.checklayer.TestLayerWithoutBackward ......... Done nnet.checklayer.TestLayerWithoutBackward __________ Test Summary: 9 Passed, 0 Failed, 0 Incomplete, 8 Skipped. Time elapsed: 0.48691 seconds.
The results show the number of passed, failed, and skipped tests. If you do not specify the 'ObservationsDimension'
option, or do not have a GPU, then the function skips the corresponding tests.
Check Multiple Observations
For multiobservation input, the layer expects an array of observations of size hbywbycbyN, where h, w, and c are the height, width, and number of channels, respectively, and N is the number of observations.
To check the layer validity for multiple observations, specify the typical size of an observation and set 'ObservationDimension'
to 4.
layer = preluLayer(20,'prelu'); validInputSize = [5 5 20]; checkLayer(layer,validInputSize,'ObservationDimension',4)
Skipping GPU tests. No compatible GPU device found. Running nnet.checklayer.TestLayerWithoutBackward .......... ... Done nnet.checklayer.TestLayerWithoutBackward __________ Test Summary: 13 Passed, 0 Failed, 0 Incomplete, 4 Skipped. Time elapsed: 0.16173 seconds.
In this case, the function does not detect any issues with the layer.
The checkLayer
function checks the validity of a custom layer by performing a series of tests.
The checkLayer
function uses these tests to check the validity of custom
intermediate layers (layers of type nnet.layer.Layer
).
Test  Description 

functionSyntaxesAreCorrect  The syntaxes of the layer functions are correctly defined. 
predictDoesNotError  predict does not error. 
forwardDoesNotError  When specified, 
forwardPredictAreConsistentInSize  When 
backwardDoesNotError  When specified, backward does not error. 
backwardIsConsistentInSize  When

predictIsConsistentInType  The outputs of 
forwardIsConsistentInType  When 
backwardIsConsistentInType  When 
gradientsAreNumericallyCorrect  When backward is specified, the gradients computed
in backward are consistent with the numerical
gradients. 
backwardPropagationDoesNotError  When backward is not specified, the derivatives
can be computed using automatic differentiation. 
The tests predictIsConsistentInType
, forwardIsConsistentInType
, and backwardIsConsistentInType
also check for GPU compatibility. To execute the layer functions on a GPU, the functions must support inputs and outputs of type gpuArray
with the underlying data type single
.
The checkLayer
function uses these tests to check the
validity of custom output layers (layers of type
nnet.layer.ClassificationLayer
or
nnet.layer.RegressionLayer
).
Test  Description 

forwardLossDoesNotError  forwardLoss does not error. 
backwardLossDoesNotError  backwardLoss does not error. 
forwardLossIsScalar  The output of forwardLoss is scalar. 
backwardLossIsConsistentInSize  When backwardLoss is specified, the output of
backwardLoss is consistent in size:
dLdY is the same size as the predictions
Y . 
forwardLossIsConsistentInType  The output of 
backwardLossIsConsistentInType  When 
gradientsAreNumericallyCorrect  When backwardLoss is specified, the gradients computed
in backwardLoss are numerically correct. 
backwardPropagationDoesNotError  When backwardLoss is not specified, the derivatives
can be computed using automatic differentiation. 
The forwardLossIsConsistentInType
and
backwardLossIsConsistentInType
tests also check for GPU compatibility. To
execute the layer functions on a GPU, the functions must support inputs and outputs of type
gpuArray
with the underlying data type single
.
To check the layer validity, the checkLayer
function generates data depending on the type of layer:
Layer Type  Description of Generated Data 

Intermediate  Values in the range [1,1] 
Regression output  Predictions and targets with values in the range [1,1] 
Classification output  Predictions with values in the range [0,1]. If
you specify the If you do
not specify the 
To check for multiple observations, specify the observation dimension using the 'ObservationDimension'
namevalue pair. If you specify the observation dimension, then the checkLayer
function checks that the layer functions are valid using generated data with minibatches of size 1 and 2. If you do not specify this namevalue pair, then the function skips the tests that check that the layer functions are valid for multiple observations.
If a test fails when you use checkLayer
,
then the function provides a test diagnostic and a framework diagnostic. The test
diagnostic highlights any issues found with the layer. The framework diagnostic provides
more detailed information.
The test functionSyntaxesAreCorrect
checks that the layer
functions have correctly defined syntaxes.
Test Diagnostic  Description  Possible Solution 

Incorrect number of input arguments for 'predict' in
Layer .  The syntax for the predict function is not
consistent with the number of layer inputs.  Specify the correct number of input and output
arguments in The syntax for [Z1,…,Zm] = predict(layer,X1,…,Xn) X1,…,Xn are the n layer inputs and
Z1,…,Zm are the m layer outputs. The values
n and m must correspond to the
NumInputs and NumOutputs properties of the
layer.TipIf the number of inputs to 
Incorrect number of output arguments for 'predict' in
Layer  The syntax for the predict function is not
consistent with the number of layer outputs.  
Incorrect number of input arguments for 'forward' in
Layer  The syntax for the optional forward function
is not consistent with the number of layer inputs.  Specify the correct number of input and output
arguments in The syntax for [Z1,…,Zm,memory] = forward(layer,X1,…,Xn) X1,…,Xn are the n layer inputs,
Z1,…,Zm are the m layer outputs, and
memory is the memory of the layer.TipIf the number of inputs to 
Incorrect number of output arguments for 'forward' in
Layer  The syntax for the optional forward function
is not consistent with the number of layer outputs.  
Incorrect number of input arguments for 'backward' in
Layer  The syntax for the optional backward function
is not consistent with the number of layer inputs and
outputs.  Specify the correct number of input and output
arguments in The syntax for [dLdX1,…,dLdXn,dLdW1,…,dLdWk] = backward(layer,X1,…,Xn,Z1,…,Zm,dLdZ1,…,dLdZm,memory)
For the outputs, TipIf the number of inputs to If the number of outputs can vary, then use TipIf the layer forward functions support 
Incorrect number of output arguments for 'backward' in
Layer  The syntax for the optional backward function
is not consistent with the number of layer outputs. 
For layers with multiple inputs or outputs, you must set the values of the layer
properties NumInputs
(or alternatively,
InputNames
) and NumOutputs
(or
alternatively, OutputNames
) in the layer constructor function,
respectively.
The checkLayer
function checks that the layer functions are
valid for single and multiple observations. To check for multiple observations, specify the observation dimension using the 'ObservationDimension'
namevalue pair. If you specify the observation dimension, then the checkLayer
function checks that the layer functions are valid using generated data with minibatches of size 1 and 2. If you do not specify this namevalue pair, then the function skips the tests that check that the layer functions are valid for multiple observations.
Test Diagnostic  Description  Possible Solution 

Skipping multiobservation tests. To enable checks with
multiple observations, specify the 'ObservationDimension'
parameter in checkLayer .  If you do not specify the
'ObservationDimension' parameter in
checkLayer , then the function skips the
tests that check data with multiple observations.  Use the command
For more information, see Layer Input Sizes. 
These tests check that the layers do not error when passed input data of valid size.
Intermediate Layers. The tests predictDoesNotError
,
forwardDoesNotError
, and
backwardDoesNotError
check that the layer functions do
not error when passed inputs of valid size. If you specify an observation
dimension, then the function checks the layer for both a single observation and
multiple observations.
Test Diagnostic  Description  Possible Solution 

The function 'predict' threw an
error:  The predict function errors when
passed data of size
validInputSize .  Address the error described in the
TipIf the layer forward functions support 
The function 'forward' threw an
error:  The optional forward function errors
when passed data of size
validInputSize .  
The function 'backward' threw an
error:  The optional backward function errors
when passed the output of
predict . 
Output Layers. The tests forwardLossDoesNotError
and
backwardLossDoesNotError
check that the layer functions
do not error when passed inputs of valid size. If you specify an observation
dimension, then the function checks the layer for both a single observation and
multiple observations.
Test Diagnostic  Description  Possible Solution 

The function 'forwardLoss' threw an
error:  The forwardLoss function errors when
passed data of size
validInputSize .  Address the error described in the
TipIf the 
The function 'backwardLoss' threw an
error:  The optional backwardLoss function
errors when passed data of size
validInputSize . 
These tests check that the layer function outputs are consistent in size.
Intermediate Layers. The test backwardIsConsistentInSize
checks that the
backward
function outputs derivatives of the correct
size.
The syntax for backward
is
[dLdX1,…,dLdXn,dLdW1,…,dLdWk] = backward(layer,X1,…,Xn,Z1,…,Zm,dLdZ1,…,dLdZm,memory)
X1,…,Xn
are the n
layer inputs
Z1,…,Zm
are the m
outputs of the layer
forward functions
dLdZ1,…,dLdZm
are the gradients backward propagated from
the next layer
memory
is the memory output of forward
if forward
is defined, otherwise, memory
is []
.
For the outputs, dLdX1,…,dLdXn
are the derivatives of the
loss with respect to the layer inputs and dLdW1,…,dLdWk
are the
derivatives of the loss with respect to the k
learnable parameters. To
reduce memory usage by preventing unused variables being saved between the forward and
backward pass, replace the corresponding input arguments with ~
.
If the number of inputs to backward
can vary, then use
varargin
instead of the input arguments after
layer
. In this case, varargin
is a cell array
of the inputs, where varargin{i}
corresponds to Xi
for i
=1,…,NumInputs
,
varargin{NumInputs+j}
and
varargin{NumInputs+NumOutputs+j}
correspond to
Zj
and dLdZj
, respectively, for
j
=1,…,NumOutputs
, and
varargin{end}
corresponds to memory
.
If the number of outputs can vary, then use varargout
instead of the
output arguments. In this case, varargout
is a cell array of the
outputs, where varargout{i}
corresponds to dLdXi
for i
=1,…,NumInputs
and
varargout{NumInputs+t}
corresponds to dLdWt
for t
=1,…,k
, where k
is the
number of learnable parameters.
The derivatives dLdX1
, …, dLdXn
must be
the same size as the corresponding layer inputs, and
dLdW1,…,dLdWk
must be the same size as the corresponding
learnable parameters. The sizes must be consistent for input data with single
and multiple observations.
Test Diagnostic  Description  Possible Solution 

Incorrect size of 'dLdX' for
'backward' .  The derivatives of the loss with respect to the layer inputs must be the same size as the corresponding layer input.  Return the derivatives

Incorrect size of the derivative of the loss
with respect to the input 'in1' for
'backward'  
The size of 'Z' returned from 'forward' must be
the same as for 'predict' .  The outputs of predict must be the
same size as the corresponding outputs of
forward .  Return the outputs 
Incorrect size of the derivative of the loss
with respect to 'W' for 'backward' .  The derivatives of the loss with respect to the learnable parameters must be the same size as the corresponding learnable parameters.  Return the derivatives

If the layer forward functions support dlarray
objects, then the software automatically determines the backward function and you do not need to specify the backward
function. For a list of functions that support dlarray
objects, see List of Functions with dlarray Support.
Output Layers. The test forwardLossIsScalar
checks that the output of the
forwardLoss
function is scalar. When the
backwardLoss
function is specified, the test
backwardLossIsConsistentInSize
checks that the outputs of
forwardLoss
and backwardLoss
are of
the correct size.
The syntax for forwardLoss
is loss
= forwardLoss(layer, Y, T)
. The input Y
corresponds to the
predictions made by the network. These predictions are the output of the previous layer. The
input T
corresponds to the training targets. The output
loss
is the loss between Y
and T
according to the specified loss function. The output loss
must be
scalar.
If the forwardLoss
function supports
dlarray
objects, then the software automatically determines
the backward loss function and you do not need to specify the
backwardLoss
function. For a list of functions that
support dlarray
objects, see List of Functions with dlarray Support.
The syntax for backwardLoss
is dLdY
= backwardLoss(layer, Y, T)
. The input Y
contains the
predictions made by the network and T
contains the training targets. The
output dLdY
is the derivative of the loss with respect to the predictions
Y
. The output dLdY
must be the same size as the layer
input Y
.
Test Diagnostic  Description  Possible Solution 

Incorrect size of 'loss' for
'forwardLoss' .  The output loss of
forwardLoss must be a scalar.  Return the output 
Incorrect size of the derivative of loss 'dLdY'
for 'backwardLoss' .  When backwardLoss is specified, the
derivatives of the loss with respect to the layer input must
be the same size as the layer input.  Return derivative If the

These tests check that the layer function outputs are consistent in type and that the layer functions are GPU compatible.
If the layer forward functions fully support dlarray
objects, then the layer
is GPU compatible. Otherwise, to be GPU compatible, the layer functions must support inputs
and return outputs of type gpuArray
.
Many MATLAB^{®} builtin functions support gpuArray
and dlarray
input arguments. For a list of
functions that support dlarray
objects, see List of Functions with dlarray Support. For a list of functions
that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).
To use a GPU for deep
learning, you must also have a CUDA^{®} enabled NVIDIA^{®} GPU with compute capability 3.0 or higher. For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).
Intermediate Layers. The tests predictIsConsistentInType
,
forwardIsConsistentInType
, and
backwardIsConsistentInType
check that the layer functions
output variables of the correct data type. The tests check that the layer
functions return consistent data types when given inputs of the data types
single
, double
, and
gpuArray
with the underlying types
single
or double
.
If you preallocate arrays using functions like
zeros
, then you must ensure that the data types of these arrays are
consistent with the layer function inputs. To create an array of zeros of the same data type of
another array, use the 'like'
option of zeros
. For
example, to initialize an array of zeros of size sz
with the same data type
as the array X
, use Z = zeros(sz,'like',X)
.
Test Diagnostic  Description  Possible Solution 

Incorrect type of 'Z' for
'predict' .  The types of the outputs
Z1,…,Zm of the
predict function must be consistent
with the inputs X1,…,Xn .  Return the outputs

Incorrect type of output 'out1' for
'predict' .  
Incorrect type of 'Z' for
'forward' .  The types of the outputs
Z1,…,Zm of the optional
forward function must be consistent
with the inputs X1,…,Xn .  
Incorrect type of output 'out1' for
'forward' .  
Incorrect type of 'dLdX' for
'backward' .  The types of the derivatives
dLdX1,…,dLdXn of the optional
backward function must be consistent
with the inputs X1,…,Xn .  Return the derivatives

Incorrect type of the derivative of the loss
with respect to the input 'in1' for
'backward' .  
Incorrect type of the derivative of loss with
respect to 'W' for 'backward' .  The type of the derivative of the loss of the learnable parameters must be consistent with the corresponding learnable parameters.  For each learnable parameter, return the derivative with the same type as the corresponding learnable parameter. 
If the layer forward functions support dlarray
objects, then the software automatically determines the backward function and you do not need to specify the backward
function. For a list of functions that support dlarray
objects, see List of Functions with dlarray Support.
Output Layers. The tests forwardLossIsConsistentInType
and
backwardLossIsConsistentInType
check that the layer
functions output variables of the correct data type. The tests check that the
layers return consistent data types when given inputs of the data types
single
, double
, and
gpuArray
with the underlying types
single
or double
.
Test Diagnostic  Description  Possible Solution 

Incorrect type of 'loss' for
'forwardLoss' .  The type of the output loss of the
forwardLoss function must be
consistent with the input Y .  Return 
Incorrect type of the derivative of loss 'dLdY'
for 'backwardLoss' .  The type of the output dLdY of the
optional backwardLoss function must be
consistent with the input Y .  Return 
If the forwardLoss
function supports dlarray
objects, then the software automatically determines the backward loss function and you do not need to specify the backwardLoss
function. For a list of functions that support dlarray
objects, see List of Functions with dlarray Support.
The test gradientsAreNumericallyCorrect
checks that the
gradients computed by the layer functions are numerically correct. The test
backwardPropagationDoesNotError
checks that the derivatives
can be computed using automatic differentiation.
Intermediate Layers. When the optional backward
function is not specified, the
test backwardPropagationDoesNotError
checks that the
derivatives can be computed using automatic differentiation. When the optional
backward
function is specified, the test
gradientsAreNumericallyCorrect
tests that the gradients
computed in backward
are numerically correct.
Test Diagnostic  Description  Possible Solution 

Expected a dlarray with no dimension labels, but
instead found labels .  When the optional backward function is
not specified, the layer forward functions must output
dlarray objects without dimension
labels.  Ensure that any dlarray objects created
in the layer forward functions do not contain dimension
labels. 
Unable to backward propagate through the layer.
Check that the 'forward' function fully supports
automatic differentiation. Alternatively, implement the
'backward' function manually .  One or more of the following:
 Check that the forward functions
support Check that the derivatives of the input
Alternatively, define a custom backward
function by creating a function named

Unable to backward propagate through the layer.
Check that the 'predict' function fully supports
automatic differentiation. Alternatively, implement the
'backward' function manually .  
The derivative 'dLdX' for 'backward' is
inconsistent with the numerical
gradient .  One or more of the following:
 If the layer forward functions support
Check that the derivatives in
If the derivatives are correctly
computed, then in the If the absolute and relative errors are within an acceptable margin of the tolerance, then you can ignore this test diagnostic. 
The derivative of the loss with respect to the
input 'in1' for 'backward' is inconsistent with the
numerical gradient .  
The derivative of loss with respect to 'W' for
'backward' is inconsistent with the numerical
gradient . 
If the layer forward functions support dlarray
objects, then the software automatically determines the backward function and you do not need to specify the backward
function. For a list of functions that support dlarray
objects, see List of Functions with dlarray Support.
Output Layers. When the optional backwardLoss
function is not specified,
the test backwardPropagationDoesNotError
checks that the
derivatives can be computed using automatic differentiation. When the optional
backwardLoss
function is specified, the test
gradientsAreNumericallyCorrect
tests that the gradients
computed in backwardLoss
are numerically correct.
Test Diagnostic  Description  Possible Solution 

Expected a dlarray with no dimension labels, but
instead found labels  When the optional backwardLoss
function is not specified, the
forwardLoss function must output
dlarray objects without dimension
labels.  Ensure that any dlarray objects created
in the forwardLoss function does not
contain dimension labels. 
Unable to backward propagate through the layer.
Check that the 'forwardLoss' function fully supports
automatic differentiation. Alternatively, implement the
'backwardLoss' function manually  One or more of the following:
 Check that the Check that the derivatives of the input
Alternatively, define a custom backward
loss function by creating a function named

The derivative 'dLdY' for 'backwardLoss' is
inconsistent with the numerical
gradient .  One or more of the following:
 Check that the derivatives in
If the derivatives are correctly
computed, then in the If the absolute and relative errors are within an acceptable margin of the tolerance, then you can ignore this test diagnostic. 
If the forwardLoss
function supports dlarray
objects, then the software automatically determines the backward loss function and you do not need to specify the backwardLoss
function. For a list of functions that support dlarray
objects, see List of Functions with dlarray Support.