Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Are TRAINBR results repeatable?

Subject: Are TRAINBR results repeatable?

From: Greg Heath

Date: 21 Feb, 2014 15:07:11

Message: 1 of 4

I was comparing TRAINBR to TRAINLM to further understand the idiosyncracies of the former. I was surprised to find, regardless of which syntax I used (e.g., sw=0 or sw=1), that

 Although designs of trainlm are repeatable, designs of trainbr appear to be unrepeatable.

 When I replace trainbr by trainlm in the following code, the results are repeatable.

clear all, close all, clc, tic
[ x, t ] = simplefit_dataset;
%plot(x,t,'LineWidth',2), hold on
for i=1:10
    net = fitnet; %default H = 10
    rng(0)
    sw = 0
    if sw == 1
        [ net tr ] = trainbr(net,x,t);
        y = net(x);
        e = t-y;
    else
        net.trainFcn = 'trainbr';
        [ net tr y e ] = train(net,x,t);% e = t-y
    end
   % plot(x,y,'ro','LineWidth',2)
    NMSE(i) = mse(e)/var(t,1) ;
end
toc % 69
NMSE

% 1e-9 * [ 0.3799 0.3558 0.3558 0.3557 0.3798 ...
% 0.2238 0.3117 0.2917 0.3118 0.3557 ]

 Any idea why?

Greg

Subject: Are TRAINBR results repeatable?

From: Greg Heath

Date: 22 Feb, 2014 07:41:08

Message: 2 of 4

"Greg Heath" <heath@alumni.brown.edu> wrote in message
> <le7q2v$4as$1@newscl01ah.mathworks.com>...
> I was comparing TRAINBR to TRAINLM to further understand
> the idiosyncracies of the former. I was surprised to find, regardless
> of which syntax I used (e.g., sw=0 or sw=1), that
>
> Although designs of trainlm are repeatable, designs of trainbr
> appear to be unrepeatable.
>
> When I replace trainbr by trainlm in the following code, the
> results are repeatable.
>
> Any ideas why?

Corrected and modified code:

clear all, close all, clc, tic
[ x, t ] = simplefit_dataset;
vart1 = var(t,1) % 8.3378
plot(x,t,'LineWidth',2), hold on
Ntrials = 5 % Fewer trials
for i=1:Ntrials
    rng(0) % Before random initial biases are assigned
    net = fitnet; %default H = 10
    sw = 0;
    if sw == 1
        [ net tr ] = trainbr(net,x,t);
        y = net(x);
        e = t-y;
    else
        net.trainFcn = 'trainbr';
        [ net tr y e ] = train(net,x,t);% e = t-y
    end
    plot(x,y,'ro','LineWidth',2)
    WB (:,i) = getwb(net); % Weights
    NMSE(i) = mse(e)/var(t,1) ; % Normalized MSE
end
NMSE %1e-10*[ 3.799 4.055 3.330 4.054 3.331 ]
RNMSE = gsqrt(NMSE) % 1e-5*[ 1.949 2.014 1.825 2.014 1.825 ]
MAEDRNMSE = mae(RNMSE(:,2:end)-RNMSE(:,1:end-1)) %1.576e-6
WB
MAEDWB = mae(WB(:,2:end)-WB(:,1:end-1)) % 1.1e-2
runtime = toc % 42 sec

The mean of the absolute value of the difference in weight estimates for
succesive trials is ~1e-2 for trainbr whereas it is only ~5e-10 for trainlm.

The mean of the absolute value of the difference in RMS errors for
succesive trials is ~2e-6 for trainbr whereas it is only ~1e-14 for trainlm.

It seems to be counter-intuitive that the trainbr code for regularization
is less robust than trainlm by ~ 4 orders ofmagnitude..

Or am I missing something?

Greg

Subject: Are TRAINBR results repeatable?

From: Greg Heath

Date: 23 Feb, 2014 00:47:11

Message: 3 of 4

"Greg Heath" <heath@alumni.brown.edu> wrote in message <le9kak$t8h$1@newscl01ah.mathworks.com>...
> "Greg Heath" <heath@alumni.brown.edu> wrote in message
> > <le7q2v$4as$1@newscl01ah.mathworks.com>...
> > I was comparing TRAINBR to TRAINLM to further understand
> > the idiosyncracies of the former. I was surprised to find, regardless
> > of which syntax I used (e.g., sw=0 or sw=1), that
> >
> > Although designs of trainlm are repeatable, designs of trainbr
> > appear to be unrepeatable.
> >
> > When I replace trainbr by trainlm in the following code, the
> > results are repeatable.
> >
> > Any ideas why?
>
> Corrected and modified code:
>
> clear all, close all, clc, tic
> [ x, t ] = simplefit_dataset;
> vart1 = var(t,1) % 8.3378
> plot(x,t,'LineWidth',2), hold on
> Ntrials = 5 % Fewer trials
> for i=1:Ntrials
> rng(0) % Before random initial biases are assigned
> net = fitnet; %default H = 10
> sw = 0;
> if sw == 1
> [ net tr ] = trainbr(net,x,t);
> y = net(x);
> e = t-y;
> else
> net.trainFcn = 'trainbr';
> [ net tr y e ] = train(net,x,t);% e = t-y
> end
> plot(x,y,'ro','LineWidth',2)
> WB (:,i) = getwb(net); % Weights
> NMSE(i) = mse(e)/var(t,1) ; % Normalized MSE
> end
> NMSE %1e-10*[ 3.799 4.055 3.330 4.054 3.331 ]
> RNMSE = gsqrt(NMSE) % 1e-5*[ 1.949 2.014 1.825 2.014 1.825 ]
> MAEDRNMSE = mae(RNMSE(:,2:end)-RNMSE(:,1:end-1)) %1.576e-6
> WB
> MAEDWB = mae(WB(:,2:end)-WB(:,1:end-1)) % 1.1e-2
> runtime = toc % 42 sec
>
> The mean of the absolute value of the difference in weight estimates for
> succesive trials is ~1e-2 for trainbr whereas it is only ~5e-10 for trainlm.
>
> The mean of the absolute value of the difference in RMS errors for
> succesive trials is ~2e-6 for trainbr whereas it is only ~1e-14 for trainlm.
>
> It seems to be counter-intuitive that the trainbr code for regularization
> is less robust than trainlm by ~ 4 orders ofmagnitude..
>
> Or am I missing something?
>
> Greg

That should be 8 orders of magnitude.

Greg

Subject: Are TRAINBR results repeatable?

From: Greg Heath

Date: 23 Jun, 2014 09:51:11

Message: 4 of 4

"Greg Heath" <heath@alumni.brown.edu> wrote in message <le7q2v$4as$1@newscl01ah.mathworks.com>...
> I was comparing TRAINBR to TRAINLM to further understand the idiosyncracies of the former. I was surprised to find, regardless of which syntax I used (e.g., sw=0 or sw=1), that
>
> Although designs of trainlm are repeatable, designs of trainbr appear to be unrepeatable.
>
> When I replace trainbr by trainlm in the following code, the results are repeatable.
>
> clear all, close all, clc, tic
> [ x, t ] = simplefit_dataset;
> %plot(x,t,'LineWidth',2), hold on
> for i=1:10
> net = fitnet; %default H = 10
> rng(0)
> sw = 0
> if sw == 1
> [ net tr ] = trainbr(net,x,t);
> y = net(x);
> e = t-y;
> else
> net.trainFcn = 'trainbr';
> [ net tr y e ] = train(net,x,t);% e = t-y
> end
> % plot(x,y,'ro','LineWidth',2)
> NMSE(i) = mse(e)/var(t,1) ;
> end
> toc % 69
> NMSE
>
> % 1e-9 * [ 0.3799 0.3558 0.3558 0.3557 0.3798 ...
> % 0.2238 0.3117 0.2917 0.3118 0.3557 ]
>
> Any idea why?

Those were 11b results. I just reran with 14a and got identical results with trainbr.

Just for the record. I'm not interested in why anymore.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us