MATLAB Examples

Visualize Features of a Convolutional Neural Network

This example shows how to visualize the features learned by convolutional neural networks.

Convolutional neural networks use features to classify images. The network learns these features itself during the training process. What the network learns during training is sometimes unclear. However, you can use the deepDreamImage function to visualize the features learned.

The convolutional layers of a network output multiple 2-D arrays. Each array (or channel) corresponds to a filter applied to the layer input. The channels output by fully connected layers correspond to high-level combinations of the features learned by earlier layers.

You can visualize what the learned features look like by using deepDreamImage to generate images that strongly activate a particular channel of the network layers.

The example requires Neural Network Toolbox™, Image Processing Toolbox™, and Neural Network Toolbox Model for AlexNet Network support package.

Contents

Load Pretrained Network

Load a pretrained AlexNet network.

net = alexnet;

Visualize Convolutional Layers

There are five 2-D convolutional layers in the AlexNet network. The convolutional layers towards the beginning of the network have a small receptive field size and learn small, low-level features. The layers towards the end of the network have larger receptive field sizes and learn larger features.

Using the Layers property, view the network architecture and locate the convolutional layers. Notice that the 2-D convolutional layers are layers 2, 6, 10, 12, and 14.

net.Layers
ans = 

  25x1 Layer array with layers:

     1   'data'     Image Input                   227x227x3 images with 'zerocenter' normalization
     2   'conv1'    Convolution                   96 11x11x3 convolutions with stride [4  4] and padding [0  0]
     3   'relu1'    ReLU                          ReLU
     4   'norm1'    Cross Channel Normalization   cross channel normalization with 5 channels per element
     5   'pool1'    Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  0]
     6   'conv2'    Convolution                   256 5x5x48 convolutions with stride [1  1] and padding [2  2]
     7   'relu2'    ReLU                          ReLU
     8   'norm2'    Cross Channel Normalization   cross channel normalization with 5 channels per element
     9   'pool2'    Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  0]
    10   'conv3'    Convolution                   384 3x3x256 convolutions with stride [1  1] and padding [1  1]
    11   'relu3'    ReLU                          ReLU
    12   'conv4'    Convolution                   384 3x3x192 convolutions with stride [1  1] and padding [1  1]
    13   'relu4'    ReLU                          ReLU
    14   'conv5'    Convolution                   256 3x3x192 convolutions with stride [1  1] and padding [1  1]
    15   'relu5'    ReLU                          ReLU
    16   'pool5'    Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  0]
    17   'fc6'      Fully Connected               4096 fully connected layer
    18   'relu6'    ReLU                          ReLU
    19   'drop6'    Dropout                       50% dropout
    20   'fc7'      Fully Connected               4096 fully connected layer
    21   'relu7'    ReLU                          ReLU
    22   'drop7'    Dropout                       50% dropout
    23   'fc8'      Fully Connected               1000 fully connected layer
    24   'prob'     Softmax                       softmax
    25   'output'   Classification Output         crossentropyex with 'tench', 'goldfish', and 998 other classes

Features on Convolutional Layer 1

Set layer to be the first convolutional layer. This layer is the second layer in the network and is named 'conv1'.

layer = 2;
name = net.Layers(layer).Name
name =

    'conv1'

Visualize the first 56 features learned by this layer using deepDreamImage by setting channels to be the vector of indices 1:56. Set 'PyramidLevels' to 1 so that the images are not scaled. To display the images together, you can use montage (Image Processing Toolbox).

deepDreamImage uses a compatible GPU, by default, if available. Otherwise it uses the CPU. A CUDA®-enabled NVIDIA® GPU with compute capability 3.0 or higher is required for training on a GPU.

channels = 1:56;

I = deepDreamImage(net,layer,channels, ...
    'PyramidLevels',1);

figure
montage(I)
title(['Layer ',name,' Features'])
|============================================|
|    Pyramid   |  Iteration   |  Activation  |
|     Level    |              |   Strength   |
|============================================|
|            1 |            1 |         2.46 |
|            1 |            2 |        38.02 |
|            1 |            3 |        73.59 |
|            1 |            4 |       109.16 |
|            1 |            5 |       144.72 |
|            1 |            6 |       180.29 |
|            1 |            7 |       215.86 |
|            1 |            8 |       251.42 |
|            1 |            9 |       286.99 |
|            1 |           10 |       322.56 |
|============================================|

These images mostly contain edges and colors, which indicates that the filters at layer 'conv1' are edge detectors and color filters. The edge detectors are at different angles, which allows the network to construct more complex features in the later layers.

Features on Convolutional Layer 2

These features are created using the features from layer 'conv1'. The second convolutional layer is named 'conv2', which corresponds to layer 6. Visualize the first 30 features learned by this layer by setting channels to be the vector of indices 1:30.

layer = 6;
channels = 1:30;

I = deepDreamImage(net,layer,channels,...
    'PyramidLevels',1);

figure
montage(I)
name = net.Layers(layer).Name;
title(['Layer ',name,' Features'])
|============================================|
|    Pyramid   |  Iteration   |  Activation  |
|     Level    |              |   Strength   |
|============================================|
|            1 |            1 |         3.86 |
|            1 |            2 |         4.87 |
|            1 |            3 |        14.14 |
|            1 |            4 |        21.64 |
|            1 |            5 |        27.33 |
|            1 |            6 |        31.88 |
|            1 |            7 |        35.66 |
|            1 |            8 |        39.32 |
|            1 |            9 |        41.96 |
|            1 |           10 |        44.27 |
|============================================|

Features on Convolutional Layers 3–5

For each of the remaining convolutional layers, visualize the first 30 features learned. To suppress detailed output on the optimization process, set 'Verbose' to 'false' in the call to deepDreamImage. Notice that the layers which are deeper into the network yield more detailed filters.

layers = [10 12 14];
channels = 1:30;

for layer = layers
    I = deepDreamImage(net,layer,channels,...
        'Verbose',false, ...
        'PyramidLevels',1);

    figure
    montage(I)
    name = net.Layers(layer).Name;
    title(['Layer ',name,' Features'])
end

Visualize Fully Connected Layers

There are three fully connected layers in the AlexNet model. The fully connected layers are towards the end of the network and learn high-level combinations of the features learned by the earlier layers.

Select the first two fully connected layers (layers 17 and 20).

layers = [17 20];

For each of these layers, use deepDreamImage to visualize the first six features. Set 'NumIterations' to 50 in the call to deepDreamImage to produce more detailed images. The images generated from the final fully connected layer correspond to the image classes.

channels = 1:6;

for layer = layers
    I = deepDreamImage(net,layer,channels, ...
        'Verbose',false, ...
        'NumIterations',50);

    figure
    montage(I)
    name = net.Layers(layer).Name;
    title(['Layer ',name,' Features'])
end

To produce images that resemble each class the most closely, select the final fully connected layer, and set channels to be the indices of the classes.

layer = 23;
channels = [9 188 231 563 855 975];

The class names are stored in the ClassNames property of the output layer (the last layer). You can view the names of the selected classes by selecting the entries in channels.

net.Layers(end).ClassNames(channels)
ans =

  1×6 cell array

  Columns 1 through 4

    'hen'    'Yorkshire terrier'    'Shetland sheepdog'    'fountain'

  Columns 5 through 6

    'theater curtain'    'geyser'

Generate detailed images that strongly activate these classes.

I = deepDreamImage(net,layer,channels, ...
    'Verbose',false, ...
    'NumIterations',50);

figure
montage(I)
name = net.Layers(layer).Name;
title(['Layer ',name,' Features'])