Matlab Shallow Network Mini Batch Training

Hello, I have been training my data through the patternnet provided by matlab and really like it's functionality and I've been seeing great results using it. I have a problem however, I want to start investigating all the functions that can be adjusted under the hood of the default patternnet, but I have such a large data size, that even though I'm connected to a cluster, my model takes about 10 hours at minimum to train. I know there are capabilities with training on the GPU but after several attempts, it says I have no memory for training. I know having a minibatch might be able to compensate for this, but I'm not entirely sure if I have to create a datastore for the minibatch to be effective. If anyone has input a minibatch into the shallow network inputs and trained on GPU, please give me some insight on the right direction to go with this. Thanks in advance.

[Nicola Galvanetto]

Did you solve your problem? I've the same...

Thanks

[Ilya Makarov]

Hi Lynn,

In my experience, it's impossible to use mini-batch training using shallow networks (patternnet) - Matlab seems to use the whole dataset as a single batch. In case you have millions of training vectors, you'll probably get an error like "Out-of-memory" or so.

There's a possibility to use an incremental training but that's not the way we wish to proceed.

In my experience the only way to go is to create a deep network for classifying sequences and then re-configure it to behave like a deep multilayer perceptron. This is how it works (at least, on my machine). Suppose you have 1M training vectors (each vector has N dimensions)

1) create a deep network for sequence classification (suppose it will have 2 hidden layers with 1024 neurons per layer, Rectified Linear Unit as an output for any hidden layer and the output classification layer with 100 classes (outputs))

layers = [...
    sequenceInputLayer(N)
    
    fullyConnectedLayer(1024)
    relyLayer
    
    fullyConnectedLayer(1024)
    relyLayer
    
    fullyConnectedLayer(100)
    softmaxLayer
    classificationLayer]

2) Choose needful training options next

options = trainingOptions(%insert your parameters here
          'MiniBatchSize',500, ...
           'MaxEpochs',100, ...
           %insert your parameters here)

Now the network we're creating will train parameters using mini-batches of 500 vectors each for 100 epochs maximum

3) prepare your data. For doing this create a cell massive inputs{1000000,1}, where the i-th cell (== inputs{i,1}) will have exactly one input training vector (the whole cell massive will have 1M cells in total)

After that create a cell array of class labels labels{1000000,1}, where the i-th cell (==labels{i,1}) will have exactly one categorical label for the given input training vector of the same index (please don't forget to make a label categorical)

4) train your network

       net=trainNetwork(inputs,labels,layers,options)

Hope it helps.

[sayak nag]

Please help I am following your advice but it seems that despite whatever I specify as my mini-batch the network is training in batch mode i.e. no of iterations per epoch is 1.

[Jingjun Liu]

The mini-batch does not work for sequenceInputLayer. That's what I found.

Answer 1

[Greg Heath]

If you have a huge dataset, it is often rewarding to just randomly divide it into m subsets. Then design with 1 and test on m-1. If the subsets are sufficiently large. it is not necessary to use m-fold cross-validation. However, you may want to design more than one net.

Hope this helps.

Thank you for formally accepting my answer

Greg