batchNormalizationLayer
Batch normalization layer
Description
A batch normalization layer normalizes a minibatch of data across all observations for each channel independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use batch normalization layers between convolutional layers and nonlinearities, such as ReLU layers.
After normalization, the layer scales the input with a learnable scale factor γ and shifts it by a learnable offset β.
Creation
Description
creates a batch normalization layer.layer
= batchNormalizationLayer
creates a batch normalization layer and sets the optional layer
= batchNormalizationLayer(Name,Value
)TrainedMean
, TrainedVariance
, Epsilon
, Parameters and Initialization, Learning Rate and Regularization, and
Name
properties using one or more namevalue pairs.
For example, batchNormalizationLayer('Name','batchnorm')
creates a batch normalization layer with the name
'batchnorm'
.
Properties
Batch Normalization
TrainedMean
— Mean statistic used for prediction
numeric vector
Mean statistic used for prediction, specified as a numeric vector of perchannel mean values.
Depending on the type of layer input, the trainNetwork
, assembleNetwork
, layerGraph
, and dlnetwork
functions automatically reshape this property to have of the following sizes:
Layer Input  Property Size 

feature input  NumChannels by1 
vector sequence input  
1D image input  1byNumChannels 
1D image sequence input  
2D image input  1by1byNumChannels 
2D image sequence input  
3D image input  1by1by1byNumChannels 
3D image sequence input 
If the BatchNormalizationStatistics
training option is 'moving'
,
then the software approximates the batch normalization statistics during training using a
running estimate and, after training, sets the TrainedMean
and
TrainedVariance
properties to the latest values of the moving
estimates of the mean and variance, respectively.
If the BatchNormalizationStatistics
training option is
'population'
, then after network training finishes, the software
passes through the data once more and sets the TrainedMean
and
TrainedVariance
properties to the mean and variance computed from
the entire training data set, respectively.
The layer uses TrainedMean
and TrainedVariance
to
normalize the input during prediction.
Data Types: single
 double
TrainedVariance
— Variance statistic used for prediction
numeric vector
Variance statistic used for prediction, specified as a numeric vector of perchannel variance values.
Depending on the type of layer input, the trainNetwork
, assembleNetwork
, layerGraph
, and dlnetwork
functions automatically reshape this property to have of the following sizes:
Layer Input  Property Size 

feature input  NumChannels by1 
vector sequence input  
1D image input  1byNumChannels 
1D image sequence input  
2D image input  1by1byNumChannels 
2D image sequence input  
3D image input  1by1by1byNumChannels 
3D image sequence input 
If the BatchNormalizationStatistics
training option is 'moving'
,
then the software approximates the batch normalization statistics during training using a
running estimate and, after training, sets the TrainedMean
and
TrainedVariance
properties to the latest values of the moving
estimates of the mean and variance, respectively.
If the BatchNormalizationStatistics
training option is
'population'
, then after network training finishes, the software
passes through the data once more and sets the TrainedMean
and
TrainedVariance
properties to the mean and variance computed from
the entire training data set, respectively.
The layer uses TrainedMean
and TrainedVariance
to
normalize the input during prediction.
Data Types: single
 double
Epsilon
— Constant to add to minibatch variances
1e5
(default)  numeric scalar
Constant to add to the minibatch variances, specified as a numeric scalar equal to or larger than 1e5
.
The layer adds this constant to the minibatch variances before normalization to ensure numerical stability and avoid division by zero.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
NumChannels
— Number of input channels
'auto'
(default)  positive integer
This property is readonly.
Number of input channels, specified as one of the following:
'auto'
— Automatically determine the number of input channels at training time.Positive integer — Configure the layer for the specified number of input channels.
NumChannels
and the number of channels in the layer input data must match. For example, if the input is an RGB image, thenNumChannels
must be 3. If the input is the output of a convolutional layer with 16 filters, thenNumChannels
must be 16.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
 char
 string
Parameters and Initialization
ScaleInitializer
— Function to initialize channel scale factors
'ones'
(default)  'narrownormal'
 function handle
Function to initialize the channel scale factors, specified as one of the following:
'ones'
– Initialize the channel scale factors with ones.'zeros'
– Initialize the channel scale factors with zeros.'narrownormal'
– Initialize the channel scale factors by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.Function handle – Initialize the channel scale factors with a custom function. If you specify a function handle, then the function must be of the form
scale = func(sz)
, wheresz
is the size of the scale. For an example, see Specify Custom Weight Initialization Function.
The layer only initializes the channel scale factors when the Scale
property is empty.
Data Types: char
 string
 function_handle
OffsetInitializer
— Function to initialize channel offsets
'zeros'
(default)  'ones'
 'narrownormal'
 function handle
Function to initialize the channel offsets, specified as one of the following:
'zeros'
– Initialize the channel offsets with zeros.'ones'
– Initialize the channel offsets with ones.'narrownormal'
– Initialize the channel offsets by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.Function handle – Initialize the channel offsets with a custom function. If you specify a function handle, then the function must be of the form
offset = func(sz)
, wheresz
is the size of the scale. For an example, see Specify Custom Weight Initialization Function.
The layer only initializes the channel offsets when the Offset
property is empty.
Data Types: char
 string
 function_handle
Scale
— Channel scale factors
[]
(default)  numeric vector
Channel scale factors γ, specified as a numeric vector.
The channel scale factors are learnable parameters. When you train a
network, if Scale
is nonempty, then
trainNetwork
uses the
Scale
property as the initial
value. If Scale
is empty, then
trainNetwork
uses the initializer specified by
ScaleInitializer
.
Depending on the type of layer input, the trainNetwork
, assembleNetwork
, layerGraph
, and dlnetwork
functions automatically reshape this property to have of the following sizes:
Layer Input  Property Size 

feature input  NumChannels by1 
vector sequence input  
1D image input  1byNumChannels 
1D image sequence input  
2D image input  1by1byNumChannels 
2D image sequence input  
3D image input  1by1by1byNumChannels 
3D image sequence input 
Data Types: single
 double
Offset
— Channel offsets
[]
(default)  numeric vector
Channel offsets β, specified as a numeric vector.
The channel offsets are learnable parameters. When you train a
network, if Offset
is nonempty, then
trainNetwork
uses the
Offset
property as the initial
value. If Offset
is empty, then
trainNetwork
uses the initializer specified by
OffsetInitializer
.
Depending on the type of layer input, the trainNetwork
, assembleNetwork
, layerGraph
, and dlnetwork
functions automatically reshape this property to have of the following sizes:
Layer Input  Property Size 

feature input  NumChannels by1 
vector sequence input  
1D image input  1byNumChannels 
1D image sequence input  
2D image input  1by1byNumChannels 
2D image sequence input  
3D image input  1by1by1byNumChannels 
3D image sequence input 
Data Types: single
 double
MeanDecay
— Decay value for moving mean computation
0.1 (default)  numeric scalar between 0
and
1
Decay value for the moving mean computation, specified as a numeric
scalar between 0
and 1
.
When the BatchNormalizationStatistics
training option is
'moving'
, at each iteration, the layer updates
the moving mean value using
$${\mu}^{*}={\lambda}_{\mu}\widehat{\mu}+(1{\lambda}_{\mu})\mu ,$$
where $${\mu}^{*}$$ denotes the updated mean, $${\lambda}_{\mu}$$ denotes the mean decay value, $$\widehat{\mu}$$ denotes the mean of the layer input, and $$\mu $$ denotes the latest value of the moving mean value.
If the BatchNormalizationStatistics
training option is
'population'
, then this option has no
effect.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
VarianceDecay
— Decay value for moving variance computation
0.1 (default)  numeric scalar between 0
and
1
Decay value for the moving variance computation, specified as a
numeric scalar between 0
and
1
.
When the BatchNormalizationStatistics
training option is
'moving'
, at each iteration, the layer updates
the moving variance value using
$${\sigma}^{2}{}^{*}={\lambda}_{{\sigma}^{2}}\widehat{{\sigma}^{2}}+(1{\lambda}_{{\sigma}^{2}}){\sigma}^{2},$$
where $${\sigma}^{2}{}^{*}$$ denotes the updated variance, $${\lambda}_{{\sigma}^{2}}$$ denotes the variance decay value, $$\widehat{{\sigma}^{2}}$$ denotes the variance of the layer input, and $${\sigma}^{2}$$ denotes the latest value of the moving variance value.
If the BatchNormalizationStatistics
training option is
'population'
, then this option has no
effect.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
Learning Rate and Regularization
ScaleLearnRateFactor
— Learning rate factor for scale factors
1
(default)  nonnegative scalar
Learning rate factor for the scale factors, specified as a nonnegative scalar.
The software multiplies this factor by the global learning rate to determine the learning rate for the scale factors in a layer. For example, if ScaleLearnRateFactor
is 2
, then the learning rate for the scale factors in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions
function.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
OffsetLearnRateFactor
— Learning rate factor for offsets
1
(default)  nonnegative scalar
Learning rate factor for the offsets, specified as a nonnegative scalar.
The software multiplies this factor by the global learning rate to determine the learning rate
for the offsets in a layer. For example, if OffsetLearnRateFactor
is 2
, then the learning rate for the offsets in the layer is twice
the current global learning rate. The software determines the global learning rate based
on the settings specified with the trainingOptions
function.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
ScaleL2Factor
— L_{2} regularization factor for scale factors
1
(default)  nonnegative scalar
L_{2} regularization factor for the scale factors, specified as a nonnegative scalar.
The software multiplies this factor by the global L_{2} regularization
factor to determine the learning rate for the scale factors in a layer. For example, if
ScaleL2Factor
is 2
, then the
L_{2} regularization for the offsets in the layer is twice the
global L_{2} regularization factor. You can specify the global
L_{2} regularization factor using the trainingOptions
function.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
OffsetL2Factor
— L_{2} regularization factor for offsets
1
(default)  nonnegative scalar
L_{2} regularization factor for the offsets, specified as a nonnegative scalar.
The software multiplies this factor by the global L_{2} regularization
factor to determine the learning rate for the offsets in a layer. For example, if
OffsetL2Factor
is 2
, then the
L_{2} regularization for the offsets in the layer is twice the
global L_{2} regularization factor. You can specify the global
L_{2} regularization factor using the trainingOptions
function.
Data Types: single
 double
 int8
 int16
 int32
 int64
 uint8
 uint16
 uint32
 uint64
Layer
Name
— Layer name
''
(default)  character vector  string scalar
Layer name, specified as a character vector or a string scalar.
For Layer
array input, the trainNetwork
,
assembleNetwork
, layerGraph
, and
dlnetwork
functions automatically assign names to layers with the name
''
.
Data Types: char
 string
NumInputs
— Number of inputs
1
(default)
This property is readonly.
Number of inputs of the layer. This layer accepts a single input only.
Data Types: double
InputNames
— Input names
{'in'}
(default)
This property is readonly.
Input names of the layer. This layer accepts a single input only.
Data Types: cell
NumOutputs
— Number of outputs
1
(default)
This property is readonly.
Number of outputs of the layer. This layer has a single output only.
Data Types: double
OutputNames
— Output names
{'out'}
(default)
This property is readonly.
Output names of the layer. This layer has a single output only.
Data Types: cell
Examples
Create Batch Normalization Layer
Create a batch normalization layer with the name 'BN1'
.
layer = batchNormalizationLayer('Name','BN1')
layer = BatchNormalizationLayer with properties: Name: 'BN1' NumChannels: 'auto' Hyperparameters MeanDecay: 0.1000 VarianceDecay: 0.1000 Epsilon: 1.0000e05 Learnable Parameters Offset: [] Scale: [] State Parameters TrainedMean: [] TrainedVariance: [] Show all properties
Include batch normalization layers in a Layer
array.
layers = [ imageInputLayer([32 32 3]) convolution2dLayer(3,16,'Padding',1) batchNormalizationLayer reluLayer maxPooling2dLayer(2,'Stride',2) convolution2dLayer(3,32,'Padding',1) batchNormalizationLayer reluLayer fullyConnectedLayer(10) softmaxLayer classificationLayer ]
layers = 11x1 Layer array with layers: 1 '' Image Input 32x32x3 images with 'zerocenter' normalization 2 '' 2D Convolution 16 3x3 convolutions with stride [1 1] and padding [1 1 1 1] 3 '' Batch Normalization Batch normalization 4 '' ReLU ReLU 5 '' 2D Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0] 6 '' 2D Convolution 32 3x3 convolutions with stride [1 1] and padding [1 1 1 1] 7 '' Batch Normalization Batch normalization 8 '' ReLU ReLU 9 '' Fully Connected 10 fully connected layer 10 '' Softmax softmax 11 '' Classification Output crossentropyex
Algorithms
Batch Normalization Layer
A batch normalization layer normalizes a minibatch of data across all observations for each channel independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use batch normalization layers between convolutional layers and nonlinearities, such as ReLU layers.
The layer first normalizes the activations of each channel by subtracting the minibatch mean and dividing by the minibatch standard deviation. Then, the layer shifts the input by a learnable offset β and scales it by a learnable scale factor γ. β and γ are themselves learnable parameters that are updated during network training.
Batch normalization layers normalize the activations and gradients propagating through a
neural network, making network training an easier optimization problem. To take full
advantage of this fact, you can try increasing the learning rate. Since the optimization
problem is easier, the parameter updates can be larger and the network can learn faster. You
can also try reducing the L_{2} and dropout regularization. With batch
normalization layers, the activations of a specific image during training depend on which
images happen to appear in the same minibatch. To take full advantage of this regularizing
effect, try shuffling the training data before every training epoch. To specify how often to
shuffle the data during training, use the 'Shuffle'
namevalue pair
argument of trainingOptions
.
The batch normalization operation normalizes the elements x_{i} of the input by first calculating the mean μ_{B} and variance σ_{B}^{2} over the spatial, time, and observation dimensions for each channel independently. Then, it calculates the normalized activations as
$$\widehat{{x}_{i}}=\frac{{x}_{i}{\mu}_{B}}{\sqrt{{\sigma}_{B}^{2}+\u03f5}},$$
where ϵ is a constant that improves numerical stability when the variance is very small.
To allow for the possibility that inputs with zero mean and unit variance are not optimal for the operations that follow batch normalization, the batch normalization operation further shifts and scales the activations using the transformation
$${y}_{i}=\gamma {\widehat{x}}_{i}+\beta ,$$
where the offset β and scale factor γ are learnable parameters that are updated during network training.
To make predictions with the network after training, batch normalization requires a fixed mean and variance to normalize the data. This fixed mean and variance can be calculated from the training data after training, or approximated during training using running statistic computations.
If the BatchNormalizationStatistics
training option is 'moving'
,
then the software approximates the batch normalization statistics during training using a
running estimate and, after training, sets the TrainedMean
and
TrainedVariance
properties to the latest values of the moving
estimates of the mean and variance, respectively.
If the BatchNormalizationStatistics
training option is
'population'
, then after network training finishes, the software
passes through the data once more and sets the TrainedMean
and
TrainedVariance
properties to the mean and variance computed from
the entire training data set, respectively.
The layer uses TrainedMean
and TrainedVariance
to
normalize the input during prediction.
Layer Input and Output Formats
Layers in a layer array or layer graph pass data to subsequent layers as formatted
dlarray
objects. The format of a dlarray
object is a
string of characters, in which each character describes the corresponding dimension of the
data. The formats consists of one or more of these characters:
"S"
— Spatial"C"
— Channel"B"
— Batch"T"
— Time"U"
— Unspecified
For example, 2D image data represented as a 4D array, where the first two dimensions
correspond to the spatial dimensions of the images, the third dimension corresponds to the
channels of the images, and the fourth dimension corresponds to the batch dimension, can be
described as having the format "SSCB"
(spatial, spatial, channel,
batch).
You can interact with these dlarray
objects in automatic differentiation
workflows such as developing a custom layer, using a functionLayer
object,
or using the forward
and predict
functions with
dlnetwork
objects.
This table shows the supported input formats of BatchNormalizationLayer
objects and
the corresponding output format. If the output of the layer is passed to a custom layer that
does not inherit from the nnet.layer.Formattable
class, or a
FunctionLayer
object with the Formattable
option set
to false
, then the layer receives an unformatted dlarray
object with dimensions ordered corresponding to the formats outlined in this table.
Input Format  Output Format 

















References
[1] Ioffe, Sergey, and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” Preprint, submitted March 2, 2015. https://arxiv.org/abs/1502.03167.
Extended Capabilities
C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.
GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.
Version History
Introduced in R2017b
Open Example
You have a modified version of this example. Do you want to open this example with your edits?
MATLAB Command
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
 América Latina (Español)
 Canada (English)
 United States (English)
Europe
 Belgium (English)
 Denmark (English)
 Deutschland (Deutsch)
 España (Español)
 Finland (English)
 France (Français)
 Ireland (English)
 Italia (Italiano)
 Luxembourg (English)
 Netherlands (English)
 Norway (English)
 Österreich (Deutsch)
 Portugal (English)
 Sweden (English)
 Switzerland
 United Kingdom (English)