Neural Network Toolbox: screen freeze when training with too many samples

Hi,

I'm training a custom feedforward network (the strucuture is rather complicated and I think doesn't matter for this question). The input 102-dimensional.

Everything works fine when I used only a few training samples. But there seems to be some magic number and when I call train(...) with more samples than this number, my screen freezes and I have to reboot my ubuntu system. This number is different depending on the training function. For trainlm it is smaller then for trainrp.

For trainrp I can use up to 1000 samples before this happens. traingd still works with more samples but I would prefer to use trainrp ...

Do you know this problem? What is causing it and how can I fix/work around it? Is there maybe some way to use minibatches that can handle more samples?

Thanks a lot! Rico

Answer 1

[Greg Heath]

Using trainrp instead of trainlm will help

Research the documentation for use of

net.efficiency.memoryreduction

Hope this helps.

Thank you for formally accepting my answer

Greg

P.S. In the near future I will submit a post on how to duplicate neural network hidden layer signals.

Answer 2

[Greg Heath]

Not enough info to give you the best advice. Some things to think about:

How many examples do you have?

How many outputs? Are they linearly independent? help cond, help corrcoef

Do you really need 102 inputs? Are they linearly independent?

Why isn't a standard single hidden layer NN sufficient? Have you tried them?

net.numWeightElements = ?

Your training set size only has to be large enough to adequately characterize the salient characteristics of the complete distribution.

You may want to randomly sample your data set for increasingly large training sets, train using 'dividetrain' and separately test with the rest of the data.

You may want to randomly sample your data set for multiple training sets, design multiple nets with 'dividetrain' and average the otputs of te multiple nets. Search the NN literature (e.g., comp.ai.neural-nets and comp.soft-sys.matlab) using word ensemble.

Hope this helps.

Greg

P.S. Using ensembles with non-custom designs may be more efficient.

[Rico]

Thanks for your answer.

Generating examples is cheap in my case. I can have as many as I want.

The net has 4 outputs. 3 are used for training, 1 is there to acces the activations of some hidden neurons (Which is what I'm actually interested in).

100 of the 102 inputs are a downscaled 10x10 image. Automatically finding out some independent sources of this signal is what the net is trying to do... net.numWeightElements = 282.

I cannot use a standard network since what I am researching is an archtecture to find a better representation of the high dimensional input (which can be found in one of the hidden layers) - it is more complicated than an autoencoder, though. Also, I should mention that I have time series data (so I cannot arbitrarily divide them without messing everything up).

Do you have an idea what is causing the sudden performance problem when I have more than 1000 examples? Is it because then not everything fits into working memory anymore or something like that? I was hoping for a simple "yeah, you shoud increase this parameter" answer .. :-)

Cheers, Rico