Documentation

This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the English verison of the page.

Note: This page has been translated by MathWorks. Please click here
To view all translated materals including this page, select Japan from the country navigator on the bottom of this page.

FullyConnectedLayer

Fully connected layer

Description

A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.

Creation

Syntax

layer = fullyConnectedLayer(outputSize)
layer = fullyConnectedLayer(outputSize,Name,Value)

Description

layer = fullyConnectedLayer(outputSize) returns a fully connected layer and specifies the OutputSize property.

example

layer = fullyConnectedLayer(outputSize,Name,Value) sets the optional properties WeightLearnRateFactor, BiasLearnRateFactor, WeightL2Factor, BiasL2Factor, and Name using name-value pairs. For example, fullyConnectedLayer(10,'Name','fc1') creates a fully connected layer with an output size of 10 and the name 'fc1'. You can specify multiple name-value pairs. Enclose each property name in single quotes.

Properties

expand all

Learning rate factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the weights in the layer. For example, if WeightLearnRateFactor is 2, then the learning rate for the weights in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

Example: 2

Learning rate factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the biases in the layer. For example, if BiasLearnRateFactor is 2, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

Example: 2

L2 regularization factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the weights in the layer. For example, if WeightL2Factor is 2, then the L2 regularization for the weights in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions function.

Example: 2

L2 regularization factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the biases in the layer. For example, if BiasL2Factor is 2, then the L2 regularization for the biases in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions function.

Example: 2

Layer name, specified as a character vector. If Name is set to '', then the software automatically assigns a name at training time.

Data Types: char

Input size for the fully connected layer, specified as a positive integer or 'auto'. If InputSize is 'auto', then the software automatically determines the input size during training.

Output size for the fully connected layer, specified as a positive integer.

Example: 10

Layer weights, specified as an OutputSize-by-InputSize matrix.

Data Types: single | double

Layer biases, specified as an OutputSize-by-1 matrix.

Data Types: single | double

Examples

expand all

Create a fully connected layer with an output size of 10 and the name 'fc1'.

layer = fullyConnectedLayer(10,'Name','fc1')
layer = 
  FullyConnectedLayer with properties:

          Name: 'fc1'

   Hyperparameters
     InputSize: 'auto'
    OutputSize: 10

   Learnable Parameters
       Weights: []
          Bias: []

  Show all properties

Include a fully connected layer in a Layer array.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,'Stride',2)
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer]
layers = 
  7x1 Layer array with layers:

     1   ''   Image Input             28x28x1 images with 'zerocenter' normalization
     2   ''   Convolution             20 5x5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   ReLU                    ReLU
     4   ''   Max Pooling             2x2 max pooling with stride [2  2] and padding [0  0  0  0]
     5   ''   Fully Connected         10 fully connected layer
     6   ''   Softmax                 softmax
     7   ''   Classification Output   crossentropyex

Create a fully connected layer with an output size of 10. Set the learning rate factor for the bias to 2. Manually initialize the weights from a Gaussian distribution with a standard deviation of 0.0001.

layers = [imageInputLayer([28 28 1])
          convolution2dLayer(5,20)
          reluLayer
          maxPooling2dLayer(2,'Stride',2)
          fullyConnectedLayer(10)
          softmaxLayer
          classificationLayer]
layers = 
  7x1 Layer array with layers:

     1   ''   Image Input             28x28x1 images with 'zerocenter' normalization
     2   ''   Convolution             20 5x5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   ReLU                    ReLU
     4   ''   Max Pooling             2x2 max pooling with stride [2  2] and padding [0  0  0  0]
     5   ''   Fully Connected         10 fully connected layer
     6   ''   Softmax                 softmax
     7   ''   Classification Output   crossentropyex

To initialize the weights of the fully connected layer, you must know the layer's input size. The input size is equal to the output size of the preceding max pooling layer, which, in turn, depends on the output size of the convolutional layer.

For one direction in a channel (feature map) of the convolutional layer, the output is . The max pooling layer has nonoverlapping regions, so it down-samples by 2 in each direction, that is, . For one channel of the convolutional layer, the output of the max pooling layer is . There are 20 channels in the convolutional layer, so the output of the max pooling layer is . This is the size of the input to the fully connected layer.

The formula for overlapping regions gives the same result. For one direction of a channel, the output is . For one channel, the output is 144. For all 20 channels in the convolutional layer, the output of the max pooling layer is 2880.

Initialize the weights of the fully connected layer from a Gaussian distribution with a mean of 0 and a standard deviation of 0.0001.

layers(5).Weights = randn([10 2880]) * 0.0001;

randn([10 2880]) returns a 10-by-2880 matrix of values from a Gaussian distribution with mean 0 and standard deviation 1. Multiplying the values by 0.0001 sets the standard deviation of the Gaussian distribution equal to 0.0001.

Initialize the biases from a Gaussian distribution with a mean of 1 and a standard deviation of 0.0001.

layers(5).Bias = randn([10 1])*0.0001 + 1;

The size of the bias vector is equal to the output size of the fully connected layer, which is 10. randn([10 1]) returns a 10-by-1 vector of values from a Gaussian distribution with a mean of 0 and a standard deviation of 1. Multiplying the values by 0.00001 sets the standard deviation of values equal to 0.00001, and adding 1 sets the mean of the Gaussian distribution equal to 1.

Algorithms

The default for the initial weights is a Gaussian distribution with mean 0 and standard deviation 0.01. The default for the initial bias is 0. For an example showing how to manually change the initialization for the weights and bias, see Specify Initial Weights and Biases in Fully Connected Layer.

Introduced in R2016a

Was this topic helpful?