Improving performance on my neural network in Matlab
3 views (last 30 days)
Show older comments
I am a novice at using neural networks, and am using them as part of a project to model the severity of traffic accidents.
I have a database in Microsoft Excel, where each row represents a different accident, and each column is a different attribute of the accident.
The attributes are the independent variables. Some of the attributes are categorical (such as weather), and some are numerical (such as Average Speed at the accident site). As per this link, I have transformed my database so that each possible value for the attribute is a separate column (ie, the accident can occur during the day or night, so I have one column in the database labled "day", which can be 1 if true, or 0 if false. One additional field is "road status", whose value could be either "Dry", "wet", "oily", "muddy", "sand/gravel", "other", or "unknown", so I created five columns which represent the first five roadway conditions, whose values can each be either 0 or 1, and if all five column's values are equal to 0, then the status is "unknown", as to maintain one degree of freedom).
I have a total of 72 columns which represent independent variables, and 6645 observations.
The dependent variables are the accident severity, either Fatal, Severe, or Light. As per the Wine Classification - MATLAB & Simulink Example (link), where there was one column for each of three different wineries, I set up three database contains three columns: "Accident Severity - Fatal", "Accident Severity - Severe" and "Accident Severity - Light". Each field can be 1 or zero, and the sum of the three fields is equal to one.
In the Matlab command window, I've specified the inputs (the 72 columns - 6645 observations, representing 6645 samples of 72 elements), and the targets (the three columns, 6645 observations, representing static data: 6645 samples of 3 elements). Next I've run the Neural Network Toolbox ("nnstart") and selected "pattern recognition", then selected my data and transposed it using "Samples are matrix rows" option, and created the neural network.
My problem is that no matter what changes I make to the number of neurons, or the division of data used for training, validation and testing, I cannot get the MSE or % Error to go down, despite numerous training attempts. I have attached a screenshot of my results.
Can anyone help me by telling me exactly what I must do in order to improve my results? I am looking to achieve a result of 90% accuracy or better.
Thank you!
0 Comments
Accepted Answer
Greg Heath
on 28 Oct 2013
I do not use shuffling. So, use this code to get a general idea.
% The code below is for net.divideFcn = 'dividetrain'. You will have to modify it if you use validation and/or test subsets.
[I N ] = size(x)
[O N ] = size(t)
Neq = N*O
Nw = (I+1)*H+(H+1)*O
Ndof = Neq-Nw
%Train the net with random initial weights
for i=1:I
xsh = x;
ind = randperm(1:N);
xsh(i,:) = x(i,ind);
% Retrain duplicate model here for option 4.
ysh = net(xsh);
SSEsh = sse(t-ysh)
MSEash = SSEsh/Ndof
NMSEash(i) = MSEash/mean(var(t'));
end
[ maxNMSEa imax ] = max(NMSEash);
% Continue training the net with x(imax,:)effectively deleted.
% Repeat I-1 timesNote:
You will have to play around with finding a good way to "effectively delete" x(imax,:) because you also have to deal with the input weights coming from the imax input node to the hidden layer.
Using [], NaN ,zeros, or mean come to mind. Do not use the shuffled version.
I would first try repmat(mean(x(imax,:),2),1,N) or, since inputs are standardized, zeros(1,N) for both x(imax,:) and IW(:,imax)
Hope this helps.
Greg
0 Comments
More Answers (3)
Greg Heath
on 25 Oct 2013
Transpose your input and target matrices
[ I N ] = size(input)
[ O N ] = size(target)
Thank you for formally accepting my answer
Greg
2 Comments
Greg Heath
on 26 Oct 2013
Edited: Greg Heath
on 26 Oct 2013
I was not aware of the samples are rows option.
[I N ] and [ O N ] are the sizes when samples are columns.
I always check those at the beginning of my command line codes.
Greg Heath
on 26 Oct 2013
With the info you have given, my only advice is to try to reduce the number of inputs.
Greg Heath
on 27 Oct 2013
The only way to guarantee an optimal reduction is to test all possible input variable combinations. However, this is computationally prohibitive. Therefore suboptimal techniques that are not as computationally prohibitive are used.
The resulting weights of any minimum MSE model are easier to understand if input AND output variables are all standardized to zero-mean/unity-variance
zx = zscore(x',1)'; zt = zscore(t',1)';
You can get a preliminary ranking of input variables using backwards STEPWISE or STEPWISEFIT to obtain a linear model.
A stepwise procedure for any model results if
1. Train multiple nets with all of the input variables
2. Keep the net with the minimum degree-of-freedom adjusted MSE, MSEa.
3. One-by-one, each variable is replaced by either
a. their mean values (zeros if standardized).
b. a random reshuffling (RANDPERM) of it's values
4. The resulting MSEa is tabulated.
5. The input corresponding to the largest MSEa is deleted
6. The net is retrained without re-initialization
7. The process is repeated until either
a. The desired no. of inputs have been deleted
b. The maximum allowable MSEa is reached
Alternate approaches:
3b. The random shuffling for each variable is performed multiple times. The mean or median MSEa is used for the ranking.
4. The net is retrained before the resulting MSEa is tabulated.
Hope this helps.
Thank you for formally accepting my answer
Greg
See Also
Categories
Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!