crossChannelNormalizationLayer

Channel-wise local response normalization layer

Description

A channel-wise local response (cross-channel) normalization layer carries out channel-wise normalization.

Creation

Syntax

layer = crossChannelNormalizationLayer(windowChannelSize)

layer = crossChannelNormalizationLayer(windowChannelSize,Name,Value)

Description

layer = crossChannelNormalizationLayer(windowChannelSize) creates a channel-wise local response normalization layer and sets the WindowChannelSize property.

layer = crossChannelNormalizationLayer(windowChannelSize,Name,Value) sets the optional properties WindowChannelSize, Alpha, Beta, K, and Name using name-value pairs. For example, crossChannelNormalizationLayer(5,'K',1) creates a local response normalization layer for channel-wise normalization with a window size of 5 and K hyperparameter 1. You can specify multiple name-value pairs. Enclose each property name in single quotes.

Properties

Cross-Channel Normalization

WindowChannelSize — Size of the channel window

positive integer

Size of the channel window, which controls the number of channels that are used for the normalization of each element, specified as a positive integer.

If WindowChannelSize is even, then the window is asymmetric. The software looks at the previous floor((w-1)/2) channels and the following floor(w/2) channels. For example, if WindowChannelSize is 4, then the layer normalizes each element by its neighbor in the previous channel and by its neighbors in the next two channels.

Example: 5

Alpha — α hyperparameter in normalization

0.0001 (default) | numeric scalar

α hyperparameter in the normalization (the multiplier term), specified as a numeric scalar.

Example: 0.0002

Beta — β hyperparameter in normalization

0.75 (default) | numeric scalar

β hyperparameter in the normalization, specified as a numeric scalar. The value of Beta must be greater than or equal to 0.01.

Example: 0.8

K — K hyperparameter in the normalization

2 (default) | numeric scalar

K hyperparameter in the normalization, specified as a numeric scalar. The value of K must be greater than or equal to 10^-5.

Example: 2.5

Layer

Name — Layer name

'' (default) | character vector | string scalar

Layer name, specified as a character vector or a string scalar. To include a layer in a layer graph, you must specify a nonempty, unique layer name. If you train a series network with the layer and Name is set to '', then the software automatically assigns a name to the layer at training time.

Data Types: char | string

NumInputs — Number of inputs

1 (default)

Number of inputs of the layer. This layer accepts a single input only.

Data Types: double

InputNames — Input names

{'in'} (default)

Input names of the layer. This layer accepts a single input only.

Data Types: cell

NumOutputs — Number of outputs

1 (default)

Number of outputs of the layer. This layer has a single output only.

Data Types: double

OutputNames — Output names

{'out'} (default)

Output names of the layer. This layer has a single output only.

Data Types: cell

Examples

Create Local Response Normalization Layer

Create a local response normalization layer for channel-wise normalization, where a window of five channels normalizes each element, and the additive constant for the normalizer $\mathit{K}$ is 1.

layer = crossChannelNormalizationLayer(5,'K',1)

layer = 
  CrossChannelNormalizationLayer with properties:

                 Name: ''

   Hyperparameters
    WindowChannelSize: 5
                Alpha: 1.0000e-04
                 Beta: 0.7500
                    K: 1

Include a local response normalization layer in a Layer array.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    crossChannelNormalizationLayer(3)
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer]

layers = 
  7x1 Layer array with layers:

     1   ''   Image Input                   28x28x1 images with 'zerocenter' normalization
     2   ''   Convolution                   20 5x5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   ReLU                          ReLU
     4   ''   Cross Channel Normalization   cross channel normalization with 3 channels per element
     5   ''   Fully Connected               10 fully connected layer
     6   ''   Softmax                       softmax
     7   ''   Classification Output         crossentropyex

Limitations

• This layer does not support 3-D image inputs or vector sequence inputs.

More About

Local Response Normalization

A channel-wise local response (cross-channel) normalization layer carries out channel-wise normalization.

This layer performs a channel-wise local response normalization. It usually follows the ReLU activation layer. This layer replaces each element with a normalized value it obtains using the elements from a certain number of neighboring channels (elements in the normalization window). That is, for each element $$x$$ in the input, trainNetwork computes a normalized value $$x^{\text{'}}$$ using

$$x^{\text{'}}=\frac{x}{{\left(K+\frac{\alpha *ss}{windowChannelSize}\right)}^{\beta }},$$

where K, α, and β are the hyperparameters in the normalization, and ss is the sum of squares of the elements in the normalization window [1]. You must specify the size of the normalization window using the windowChannelSize argument of the crossChannelNormalizationLayer function. You can also specify the hyperparameters using the Alpha, Beta, and K name-value pair arguments.

The previous normalization formula is slightly different than what is presented in [1]. You can obtain the equivalent formula by multiplying the alpha value by the windowChannelSize.

References

[1] Krizhevsky, A., I. Sutskever, and G. E. Hinton. "ImageNet Classification with Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems. Vol 25, 2012.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.