This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the English version of the page.

Note: This page has been translated by MathWorks. Click here to see
To view all translated materials including this page, select Country from the country navigator on the bottom of this page.

Use parfeval to Train Multiple Deep Learning Networks

This example shows how to use parfeval for a parameter sweep on the depth of the network architecture. Deep Learning training often takes hours or days, and searching for good architectures can be difficult. You can use parallel computing to speed up and automate your search for good models. If you have access to a machine with multiple GPUs, you can run this script on a local copy of the data set with a local parpool. If you want to use more resources, you can scale up deep learning training to the cloud. This example shows how to use parfeval to perform a parameter sweep on the depth of a network architecture in a cluster in the cloud. Using parfeval allows you to train in the background without blocking MATLAB, and provides options to stop early if results are good enough. You can modify this script to do a parameter sweep on any other parameter. Also, this example shows how to obtain feedback from the workers during computation using a DataQueue.

Before you can run this example, you need to configure a cluster and upload your data to the Cloud. To get started with the cloud, set up Cloud Center, link it to an Amazon Web Services (AWS) account and create a cluster. For instructions, see Getting Started with Cloud Center. After that, upload your data to an Amazon S3 bucket and use it directly from MATLAB. For an example, see Upload Deep Learning Data to the Cloud.

Load the Data Set from the Cloud

Load the train and test data set from the cloud using imageDatastore. Split the training data set into training and validation and keep the test data set to test the best network from the parameter sweep. This example shows how to use a copy of CIFAR-10 data that is already stored in Amazon S3. To ensure that the workers have access to the datastore in the cloud, make sure that the environment variables for the AWS credentials have been set correctly. For instructions, see Upload Deep Learning Data to the Cloud.

imds = imageDatastore('s3://cifar10cloud/cifar10/train', ...
 'IncludeSubfolders',true, ...
 'LabelSource','foldernames');

imdsTest = imageDatastore('s3://cifar10cloud/cifar10/test', ...
 'IncludeSubfolders',true, ...
 'LabelSource','foldernames');

[imdsTrain,imdsValidation] = splitEachLabel(imds,0.9);

To train the network with augmented image data, create an augmentedImageDatastore object. Use random translations and horizontal reflections. Data augmentation helps prevent the network from overfitting and memorizing the exact details of the training images.

imageSize = [32 32 3];
pixelRange = [-4 4];
imageAugmenter = imageDataAugmenter( ...
    'RandXReflection',true, ...
    'RandXTranslation',pixelRange, ...
    'RandYTranslation',pixelRange);
augmentedImdsTrain = augmentedImageDatastore(imageSize,imdsTrain, ...
    'DataAugmentation',imageAugmenter, ...
    'OutputSizeMode','randcrop');

Train Several Networks Simultaneously

Define the training options. Set the mini-batch size and scale the initial learning rate linearly according to the mini-batch size. Set the validation frequency so that trainNetwork validates the network once per epoch.

miniBatchSize = 128;
initialLearnRate = 1e-1 * miniBatchSize/256;
validationFrequency = floor(numel(imdsTrain.Labels)/miniBatchSize);
options = trainingOptions('sgdm', ...
    'MiniBatchSize',miniBatchSize, ... % Set the mini-batch size
    'Verbose',false, ... % Do not send command line output.
    'InitialLearnRate',initialLearnRate, ... % Set the scaled learning rate.
    'L2Regularization',1e-10, ...
    'MaxEpochs',40, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationPatience',Inf, ...
    'ValidationFrequency', validationFrequency, ...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropFactor',0.1, ...
    'LearnRateDropPeriod',35);

Choose the depths for the network architecture to do a parameter sweep on. Perform a parallel parameter sweep training several networks simultaneously using parfeval. Use a loop to iterate through the different network architectures in the sweep. Create a helper function, createNetworkArchitecture, that takes an input argument to control the depth of the network and creates an architecture for CIFAR-10. Use parfeval to offload the computations performed by trainNetwork to a worker in the cluster. parfeval returns a future variable to hold the trained networks when computations are done. To obtain feedback from the workers in the form of training progress plots, see Send Feedback Data During Training below.

netDepths = 1:4;
for idx = 1:numel(netDepths)
    networksFuture(idx) = parfeval(@trainNetwork, 1, ...
        augmentedImdsTrain, createNetworkArchitecture(netDepths(idx)), options);
end

parfeval does not block MATLAB and you can continue executing commands. You can use fetchNext to wait for the next network to complete training and use it while the rest of the networks are still training. For example, you can check its accuracy and, if it is over a desired threshold, cancel the rest of the futures by invoking cancel on the future array. You can also use the afterEach function to automatically invoke a function on the trained network immediately after it finishes training. The following code computes the accuracy immediately after each of the networks finish and while the rest are still training. To obtain the accuracy of the network, classify with the validation data set and compare the predicted labels against the labels in the validation data set.

accuraciesFuture = afterEach(networksFuture, @(network) mean(classify(network,imdsValidation) == imdsValidation.Labels), 1);

To obtain the trained networks and their accuracies, use fetchOutputs on networksFuture and accuraciesFuture. fetchOutputs waits until the futures finish their computations. To obtain a cell array output, set the name-value pair argument UniformOutput to false.

trainedNetworks = fetchOutputs(networksFuture, 'UniformOutput', false)
accuracies = fetchOutputs(accuraciesFuture)
trainedNetworks =

  4×1 cell array

    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}


accuracies =

    0.7540
    0.8000
    0.8054
    0.8060

Select the best network in terms of accuracy. Test its performance against the test data set.

[~, I] = max(accuracies);
bestNetwork = trainedNetworks{I(1)};
YPredicted = classify(bestNetwork,imdsTest);
accuracy = sum(YPredicted == imdsTest.Labels)/numel(imdsTest.Labels)
accuracy =

    0.8033

Calculate the confusion matrix for the test data and display it as a heatmap.

figure
[cmat,classNames] = confusionmat(imdsTest.Labels,YPredicted);
h = heatmap(classNames,classNames,cmat);
xlabel('Predicted Class');
ylabel('True Class');
title('Confusion Matrix');

Send Feedback Data During Training

Prepare and initialize the plots that will show the training progress in each of the workers. Use animatedLine for a convenient way to show changing data.

f = figure;
for i=1:4
    subplot(2,2,i)
    xlabel('Iteration');
    ylabel('Training accuracy');
    lines(i) = animatedline;
end

Use a DataQueue to send the training progress data from the workers to the client and plot it. Use afterEach to update the plots each time training progress feedback is sent from the workers. The parameter opts will contain information about the worker, training iteration and training accuracy.

D = parallel.pool.DataQueue;
afterEach(D, @(opts) updatePlot(lines, opts{:}));

Choose the depths for the network architecture to do a parameter sweep on and perform the parallel parameter sweep using parfeval. To allow the workers to access any helper function in this script, add the script to the current pool as an attached file. Define an output function in the training options to send the training progress from the workers to the client. The training options depend on the index of the worker and they need to be included inside the for loop.

netDepths = 1:4;
addAttachedFiles(gcp,mfilename);
for idx = 1:numel(netDepths)

    miniBatchSize = 128;
    initialLearnRate = 1e-1 * miniBatchSize/256; % Scale the learning rate according to the mini-batch size.
    validationFrequency = floor(numel(imdsTrain.Labels)/miniBatchSize);

    options = trainingOptions('sgdm', ...
        'OutputFcn',@(state) sendTrainingProgress(D,idx,state), ... % Set an output function to send intermediate results to the client.
        'MiniBatchSize',miniBatchSize, ... % Set the corresponding MiniBatchSize in the sweep.
        'Verbose',false, ... % Do not send command line output.
        'InitialLearnRate',initialLearnRate, ... % Set the scaled learning rate.
        'L2Regularization',1e-10, ...
        'MaxEpochs',40, ...
        'Shuffle','every-epoch', ...
        'ValidationData',imdsValidation, ...
        'ValidationPatience',Inf, ...
        'ValidationFrequency', validationFrequency, ...
        'LearnRateSchedule','piecewise', ...
        'LearnRateDropFactor',0.1, ...
        'LearnRateDropPeriod',35);

    networksFuture(idx) = parfeval(@trainNetwork, 1, ...
        augmentedImdsTrain, createNetworkArchitecture(netDepths(idx)), options);
end

Use afterEach to automatically classify and obtain the accuracy of each network when they finish training.

accuraciesFuture = afterEach(networksFuture, @(network) mean(classify(network,imdsValidation) == imdsValidation.Labels), 1);

Fetch the trained networks and accuracies invoking fetchOutputs on the future variables.

trainedNetworks = fetchOutputs(networksFuture, 'UniformOutput', false)
accuracies = fetchOutputs(accuraciesFuture)
trainedNetworks =

  4×1 cell array

    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}
    {1×1 SeriesNetwork}


accuracies =

    0.7540
    0.8000
    0.8054
    0.8060

Helper Functions

Define a network architecture for CIFAR-10. To simplify the code, use convolutional blocks containing several convolutional layers that convolve the input. The pooling layers downsample the spatial dimensions. Use an input argument to control the depth of the network.

function layers = createNetworkArchitecture(netDepth)
    imageSize = [32 32 3];
    netWidth = round(16/sqrt(netDepth)); % netWidth controls the number of filters in a convolutional block

    layers = [
        imageInputLayer(imageSize)

        convolutionalBlock(netWidth,netDepth)
        maxPooling2dLayer(2,'Stride',2)
        convolutionalBlock(2*netWidth,netDepth)
        maxPooling2dLayer(2,'Stride',2)
        convolutionalBlock(4*netWidth,netDepth)
        averagePooling2dLayer(8)

        fullyConnectedLayer(10)
        softmaxLayer
        classificationLayer
    ];
end

Define a function to make easier to create convolutional blocks in the network architecture.

function layers = convolutionalBlock(numFilters,numConvLayers)
    layers = [
        convolution2dLayer(3,numFilters,'Padding','same')
        batchNormalizationLayer
        reluLayer
    ];

    layers = repmat(layers,numConvLayers,1);
end

Define a function to send the training progress to the client through a DataQueue.

function sendTrainingProgress(D,idx,info)
    if info.State == "iteration"
        send(D,{idx,info.Iteration,info.TrainingAccuracy});
    end
end

Define an update function to update the plots when a worker sends an intermediate result.

function updatePlot(lines,idx,iter,acc)
    addpoints(lines(idx),iter,acc);
    drawnow limitrate nocallbacks
end

See Also

| | | |

Related Topics