Thread Subject: Backpropagation Neural network Code problem

I am working on my thesis on face recognition on features of face, using backpropagation neural network. For it I had creating my own code for it (I am not using the building function). But the weights did not converge. Can someone please look at the code and hint me what might be wrong with the code.
From Adeel Raza

"Adeel " <neoresearcher@gmail.com> wrote in message <gpfjg6$t0n$1@fred.mathworks.com>...
> I am working on my thesis on face recognition on features of face, using backpropagation neural network. For it I had creating my own code for it (I am not using the building function). But the weights did not converge. Can someone please look at the code and hint me what might be wrong with the code.
> From Adeel Raza

I am sorry that I forgot to add my code in the previous posting

Here is the code

function [v,v0,w,w0] = trainBPElementWise(v,v0,w,w0)
%% ==Step_0==Initialization=================================
clc
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    e = 0.05; % error limit
    n = 2; %INPUT
    p = 4;
    m = 1; % OUTPUT
    alpha = 0.2;
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    cd('E:\thesisExtra\nnet')
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    trainRecords = size(x,1);
    t = [0; 1; 1; 0];
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randc(n,p);
        w = randc(p,m);
        w0 = randc(1,m);
        v0 = rand(1,p);
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    dinj = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Second hidden Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    dk = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;

%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('Epoch : %4g, max err : %d',iteration, errorMax(iteration)));
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Second Layer
        for k=1:m
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = (2/(1+exp(-yin(Tp,k))))-1; %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        dk(Tp,k) = (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * dk(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * dk(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            for k=1:m
                dinj(Tp,j) = dinj(Tp,j) + dk(Tp,k) * w(j,k);
            end
%%% Error Info = dinj * (Derivative of First layer activation function)
           dj(Tp,j) = (dinj(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
    if (~mod(iteration,200))
        figure
        plot(1:200,errorMax(iteration-199:iteration))
        getframe;% F(iteration) = getframe; movie(F);
    end
    iteration = iteration +1;
    if (iteration > 600)
        break
    end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
clf('reset'), cla reset, plot(errorMax)
xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

On Mar 14, 2:49=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> "Adeel " <neoresearc...@gmail.com> wrote in message <gpfjg6$t0...@fred.ma=
thworks.com>...
> > I am working on my thesis on face recognition on features of face, usin=
g backpropagation neural network. For it I had creating my own code for it =
(I am not using the building function). But the weights did not converge. C=
an someone please look at the code and hint me what might be wrong with the=
 code.
> > From Adeel Raza
>
> I am sorry that I forgot to add my code in the previous posting
>
> Here is the code
>
> function [v,v0,w,w0] =3D trainBPElementWise(v,v0,w,w0)
> %% =A0=3D=3DStep_0=3D=3DInitialization=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> clc
> %%% Enter the Architecture detail
> =A0 =A0 disp('Enter the Architecture detail');
> =A0 =A0 e =3D 0.05; % error limit
> =A0 =A0 n =3D 2; =A0%INPUT
> =A0 =A0 p =3D 4;
> =A0 =A0 m =3D 1; % OUTPUT
> =A0 =A0 alpha =3D 0.2;
> % =A0-----------------------------------------------------------
> %%% Initialization of the Input / Target
> =A0 =A0 cd('E:\thesisExtra\nnet')
> =A0 =A0 disp('Loading the input vector x and target vectors')
> =A0 =A0 x =3D [0 0; 0 1; 1 0; 1 1];
> =A0 =A0 trainRecords =3D size(x,1);
> =A0 =A0 t =3D [0; 1; 1; 0];
> % =A0----------------------------------------------------------
> %%% Initialization of the weights
> if (nargin =A0< 2)
> =A0 =A0 disp('weights v and w are getting initialised randomly');
> =A0 =A0 =A0 =A0 v =3Drandc(n,p);
> =A0 =A0 =A0 =A0 w =A0=3Drandc(p,m);
> =A0 =A0 =A0 =A0 w0 =3Drandc(1,m);
> =A0 =A0 =A0 =A0 v0 =3D rand(1,p); =A0 =A0 =A0 =A0
> end
> %%%First hidden Layer
> =A0 =A0 zin =3D zeros(trainRecords,p);
> =A0 =A0 z =3D =A0zeros(trainRecords,p);
> =A0 =A0 dinj =3D zeros(trainRecords,p);
> =A0 =A0 dj =3D =A0zeros(trainRecords,p);
> =A0 =A0 chv =3D zeros(n,p);
> =A0 =A0 chv0 =3D zeros(1,p);
> %%%Second hidden Layer
> =A0 =A0 yin =3D zeros(trainRecords,m);
> =A0 =A0 y =3D zeros(trainRecords,m);
> =A0 =A0 dk =3D zeros(trainRecords,m);
> =A0 =A0 chw =3D zeros(p,m);
> =A0 =A0 chw0 =3D zeros(1,m);
> iteration =3D1;
> er =3D 0; error =3D 0;
>
> %% =A0=3D=3DStep_1=3D=3DWhile_stoping_condition_is_false=3D=3Ddo_step_2-9=
=3D=3D=3D
> while er=3D=3D0
> =A0 =A0 errorMax(iteration) =3D max(max(error));
> =A0 =A0 disp(sprintf('Epoch : %4g, max err : %d',iteration, errorMax(iter=
ation)));
> %%% =3D=3DStep_2=3D=3DFor_Each_Training_pair=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3Ddo_Steps_3-8=3D=3D=3D
> =A0 =A0 for Tp=3D1:trainRecords
> %% Feed forward: =A0 =A0 =A0
> %%% =3D=3DStep_3=3D=3DX_get_Input=3D=3D=3D=3D=3DAlready_done=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> %%% =3D=3DStep_4=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> %%% First Layer
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zin(Tp,j) =3D zin(Tp,j) + x(Tp,i) * v(i,j=
);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 zin(Tp,j) =3D v0(j) + zin(Tp,j);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0z(Tp,j) =3D (2/(1+exp(-zin(Tp,j))))-1; %activa=
tion function
> =A0 =A0 =A0 =A0 end
> %%% =3D=3DStep_5=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 %%% Second Layer
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 yin(Tp,k) =3D yin(Tp,k) + z(Tp,j) * w(j,k=
);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 yin(Tp,k) =3D w0(k) + yin(Tp,k);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0y(Tp,k) =3D (2/(1+exp(-yin(Tp,k))))-1; %activa=
tion function
> =A0 =A0 =A0 =A0 end

0. If you want someone to proofread your code, please
have the coutesy to insert copious comments.
1. Use randn to initialize weights and thresholds with
normally distributed real values.
2. You only have 1 hidden layer the second layer is the
output layer.
3. Use matrix multiplication instead of loops
4. Center inputs using bipolar binary coding
(i.e., {-1,1} )to efficiently use your odd hidden
layer tanh activation function
5. It is not necessary to use a nonlinear
output activation function. However, if you do,
it is more efficient to use the logarithmic
sigmoid for unipolar binary output coding and
the tanh for bipolar binary output coding.

That's all for now.

Hope this helps.

Greg

> %% Backpropagation of error (Training Started)
> %%% =3D=3DStep_6=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 %%% Error Info =3D (t - y) * (Derivative of Second layer activati=
on function)
> =A0 =A0 =A0 =A0 dk(Tp,k) =3D (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(=
1 - y(Tp,k)));
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 chw(j,k) =3D alpha * dk(Tp,k) * z(Tp,j);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 chw0(k) =3D alpha * dk(Tp,k);
> =A0 =A0 =A0 =A0 end
> %%% =3D=3DStep_7=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 dinj(Tp,j) =3D dinj(Tp,j) + dk(Tp,k) * w(=
j,k);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> %%% =A0 =A0 Error Info =3D dinj * (Derivative of First layer activation f=
unction)
> =A0 =A0 =A0 =A0 =A0 =A0dj(Tp,j) =3D (dinj(Tp,j) * ((1/2) * (1 + z(Tp,j))*=
 (1 - z(Tp,j))));
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 chv(i,j) =3D alpha * dj(Tp,j) * x(Tp,i);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 chv0(j) =3D alpha * dj(Tp,j);
> =A0 =A0 =A0 =A0 end
> %% =3D=3DStep_8=3D=3DUpdate_weights_and_biases=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 w(j,k)=3Dw(j,k)+chw(j,k);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0w0(k)=3Dw0(k)+chw0(k);
> =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 v(i,j)=3Dv(i,j)+chv(i,j);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0v0(j)=3Dv0(j)+chv0(j);
> =A0 =A0 =A0 =A0 end
> =A0 =A0 end
> %% =A0=3D=3DStep_9=3D=3DTest_stoping_condition=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =A0 =A0 error =3D sqrt((t-y).^2);
> =A0 =A0 %%% Update the stopping condition variable
> =A0 =A0 if max(max(error)) < e % W/P =3D e, W =3Dweights,P =3Dinputs, e =
=3D errors
> =A0 =A0 =A0 =A0 er =3D1;
> =A0 =A0 else
> =A0 =A0 =A0 =A0 er =3D 0;
> =A0 =A0 end
> =A0 =A0 if (~mod(iteration,200))
> =A0 =A0 =A0 =A0 figure
> =A0 =A0 =A0 =A0 plot(1:200,errorMax(iteration-199:iteration))
> =A0 =A0 =A0 =A0 getframe;% =A0 =A0 F(iteration) =3D getframe; movie(F);
> =A0 =A0 end
> =A0 =A0 iteration =3D iteration +1;
> =A0 =A0 if (iteration > 600)
> =A0 =A0 =A0 =A0 break
> =A0 =A0 end
> end %% End of wile loop Step 1
> save('weight6040.dat','v','v0','w','w0')
> clf('reset'), cla reset, plot(errorMax)
> xlabel('iteration '), ylabel('error '), title('Plot of the error')
> end %% End of Function / File

"Adeel "

Sir
1. OK i will insert much more comments in the code.

2. You are right that the other layer is output layer, But I had two layers, because output layer had weights (plus bias) on them. Not to mention the activation function etc.

3. The name of this file is element wise, that is element by element every step, Just a precaution to safe guard against any malfunction or mistake in the matrix multiplication done by me. (Although i had created one file using matrix multiplication).

4. OK i agree i did not know that. Plus i did not know what "logarithmic sigmoid" is. Are u talking about bipolar sigmoid. ( I have not used tanh, I had used bipolar sigmoid.). In the book "Fundamentals of neural networks by laurene Fausett of pearson education" They had used for both layer the bipolar sigmoid. and according to the text they had converge the weight using the bipolar sigmoid as an activation function on both the layers. (unless i had understood it incorrectly. According to them they had solve the xor problem solution in binary representation, to get converge in 3000 epochs.. Mine never did).

On Mar 17, 2:40 pm, Greg Heath <he...@alumni.brown.edu> wrote:
> On Mar 16, 11:46 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> > 1. OK i will insert much more comments in the code.
>
> > 2. You are right that the other layer is output layer, But I had two la=
yers, because output layer had weights (plus bias) on them. Not to mention =
the activation function etc.

Two important points on terminology:

1. The input layer is not a neuron layer.
2. The output layer is not a hidden layer. (correct your comment in
the code)

> > 3. The name of this file is element wise, that is element by element ev=
ery step, Just a precaution to safe guard against any malfunction or mistak=
e in the matrix multiplication done by me. (Although i had created one file=
 using matrix multiplication).
>
> > 4. OK i agree i did not know that. Plus i did not know what "logarithmi=
c sigmoid" is. Are u talking about bipolar sigmoid. ( I have not used tanh,=
 I had used bipolar sigmoid.). In the book "Fundamentals ofneuralnetworks b=
y laurene Fausett of pearson education" They had used for both layer the bi=
polar sigmoid. and according to the text they had converge the weight using=
 the bipolar sigmoid as an activation function on both the layers. (unless =
i had understood it incorrectly. According to them they had solve the xor p=
roblem solution in binary representation, to get converge in 3000 epochs.. =
Mine never did)

logsig(x) =3D 1/(1+exp(-x))
tanh(x/2) =3D 2*logsig(x)-1

You might be interested in

http://groups.google.com/group/comp.ai.neural-nets/
msg/1c33c39b186dd026?hl=3Den

Note the learning times using the batch L-M algorithm.

Hope this helps.

Greg

On Mar 17, 3:13=A0pm, Greg Heath <he...@alumni.brown.edu> wrote:
> On Mar 17, 2:40 pm, Greg Heath <he...@alumni.brown.edu> wrote:
>
> > On Mar 16, 11:46 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> > > 1. OK i will insert much more comments in the code.
>
> > > 2. You are right that the other layer is output layer, But I had two =
layers, because output layer had weights (plus bias) on them. Not to mentio=
n the activation function etc.
>
> Two important points on terminology:
>
> 1. The input layer is not a neuron layer.
> 2. The output layer is not a hidden layer. (correct your comment in
> the code)
>
> > > 3. The name of this file is element wise, that is element by element =
every step, Just a precaution to safe guard against any malfunction or mist=
ake in the matrix multiplication done by me. (Although i had created one fi=
le using matrix multiplication).
>
> > > 4. OK i agree i did not know that. Plus i did not know what "logarith=
mic sigmoid"

WHOOPS! I meant logistic sigmoid. See below

> > > is. Are u talking about bipolar sigmoid.

No. Bipolar sigmoid is a scaled tanh. See beow.

( I have not used tanh, I had used bipolar sigmoid.). In the book
"Fundamentals ofneuralnetworks by laurene Fausett of pearson
education" They had used for both layer the bipolar sigmoid. and
according to the text they had converge the weight using the bipolar
sigmoid as an activation function on both the layers. (unless i had
understood it incorrectly. According to them they had solve the xor
problem solution in binary representation, to get converge in 3000
epochs.. Mine never did)
>
> logsig(x) =3D 1/(1+exp(-x))

Logistic Sigmoid

> tanh(x/2) =3D 2*logsig(x)-1
>
> You might be interested in
>
> http://groups.google.com/group/comp.ai.neural-nets/
> msg/1c33c39b186dd026?hl=3Den
>
> Note the learning times using the batch L-M algorithm.

Also

http://groups.google.com/group/comp.ai.neural-nets/browse_thread/thread/a44=
4b0a03ffcdafb?hl=3Den&tvc=3D2&q=3Dgreg-heath+backpropagation+tutorial+

Hope this helps.

Greg

http://www.mathworks.com/matlabcentral/newsreader/create_message?reply_id=635662

Greg Heath <heath@alumni.brown.edu> wrote in message

<dae52b80-1b2c-436a-9dd5-d6127ab8b97b@j35g2000yqh.googlegroups.com>...
> On Mar 17, 3:13=A0pm, Greg Heath <he...@alumni.brown.edu> wrote:
> > On Mar 17, 2:40 pm, Greg Heath <he...@alumni.brown.edu> wrote:
> >
> > > On Mar 16, 11:46 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> > > > 1. OK i will insert much more comments in the code.
> >
> > > > 2. You are right that the other layer is output layer, But I had two =
> layers, because output layer had weights (plus bias) on them. Not to mentio=
> n the activation function etc.
> >
> > Two important points on terminology:
> >
> > 1. The input layer is not a neuron layer.
> > 2. The output layer is not a hidden layer. (correct your comment in
> > the code)
> >
> > > > 3. The name of this file is element wise, that is element by element =
> every step, Just a precaution to safe guard against any malfunction or mist=
> ake in the matrix multiplication done by me. (Although i had created one fi=
> le using matrix multiplication).
> >
> > > > 4. OK i agree i did not know that. Plus i did not know what "logarith=
> mic sigmoid"
>
> WHOOPS! I meant logistic sigmoid. See below
>
> > > > is. Are u talking about bipolar sigmoid.
>
> No. Bipolar sigmoid is a scaled tanh. See beow.
>
> ( I have not used tanh, I had used bipolar sigmoid.). In the book
> "Fundamentals ofneuralnetworks by laurene Fausett of pearson
> education" They had used for both layer the bipolar sigmoid. and
> according to the text they had converge the weight using the bipolar
> sigmoid as an activation function on both the layers. (unless i had
> understood it incorrectly. According to them they had solve the xor
> problem solution in binary representation, to get converge in 3000
> epochs.. Mine never did)
> >
> > logsig(x) =3D 1/(1+exp(-x))
>
> Logistic Sigmoid
>
> > tanh(x/2) =3D 2*logsig(x)-1
> >
> > You might be interested in
> >
> > http://groups.google.com/group/comp.ai.neural-nets/
> > msg/1c33c39b186dd026?hl=3Den
> >
> > Note the learning times using the batch L-M algorithm.
>
> Also
>
> http://groups.google.com/group/comp.ai.neural-nets/browse_thread/thread/a44=
> 4b0a03ffcdafb?hl=3Den&tvc=3D2&q=3Dgreg-heath+backpropagation+tutorial+
>
> Hope this helps.
>
> Greg

Sir.
My basic and only concern is "Why did my algo did not converge ?"

Is binary sigmoid is logistic sigmoid ( logsig(x) ) ?

You are right. Input layer is not included in the layer count. (which i also didnot counted). But i

counted output layer because and hidden layer (the layer next to input layer) because both those

layer had weight which did change in the learning process.. Secondly the text book also called it

two layer. Another text book "Neural network. Algorithums, applications, and programming techniques

by James A. Freeman / David M. Skapura" also called it two layer backpropagation network. If i had

understood it incorrectly I would be greatful if u correct me.
I think in matlab documentation they didnot count the output layer. (i will conform that when i

check that) if u r saying according to that then i agree with u.

the last link that you send
http://groups.google.com/group/comp.ai.neural-nets/browse_thread/thread/a44=4b0a03ffcdafb?hl=3Den&tv

c=3D2&q=3Dgreg-heath+backpropagation+tutorial+
or
http://groups.google.com/group/comp.ai.neural-nets/browse_thread/thread/a44=
didnot render a page the responce was
Topic not found
"We're sorry, but we were unable to find the topic you were looking for. Perhaps the URL you clicked

on is out of date or broken? "

I am sending u the flowchart of the algo at the email addres of u's at gmail
at heath@alumni.brown.edu

I checked the matlab documentation they did count the output layer. Although, I did comment the layer as hidden layer instead of output layer. And u r right to change that to output layer.

Greg Heath <heath@alumni.brown.edu>
1. No. Bipolar sigmoid is a scaled tanh.

2. logsig(x) =3D 1/(1+exp(-x))
Logistic Sigmoid
3. tanh(x/2) =3D 2*logsig(x)-1

Sir.
I categorically disagree with the statement 1, 2, and 3.
1. Bipolar sigmoid is not a scaled tanh (hyperpolic tangent). Because according to (Fausett 1994)
Binary sigmoid:
f(x) = 1/(1 + exp(-x))
and “logistic sigmoid can be scaled Bipolar to have any range of values that is appropriate for a given problem. The most common range is from -1 to 1; we call this sigmoid the bipolar sigmoid.
g(x) = 2 / (1 + exp(-x)) - 1
”
and according to (Fausett 1994) also “The hyperbolic tangent is

h(x) = (exp(x) – exp(-x)) / (exp(x) + exp(-x)) “ at page number 37.

Fausett, L. (1994). Common Activation Functions. Fumdamentals of neural networks Architectures, algorithms, and applications, Pearson education (Singapore) Pte. Ltd., Indian Branch, 482 F.I.E. Patparganj, Delhi 110 092, India: 467.

On Mar 18, 4:29=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> http://www.mathworks.com/matlabcentral/newsreader/create_message?repl...
>
> Greg Heath <he...@alumni.brown.edu> wrote in message
> <dae52b80-1b2c-436a-9dd5-d6127ab8b...@j35g2000yqh.googlegroups.com>...
> > On Mar 17, 3:13=3DA0pm, Greg Heath <he...@alumni.brown.edu> wrote:
> > > On Mar 17, 2:40 pm, Greg Heath <he...@alumni.brown.edu> wrote:
> > > > On Mar 16, 11:46 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> > > > > 1. OK i will insert much more comments in the code.
> > > > > 2. You are right that the other layer is output layer, But I had =
two =3D
> > layers, because output layer had weights (plus bias) on them. Not to me=
ntio=3D
> > n the activation function etc.
>
> > > Two important points on terminology:
>
> > > 1. The input layer is not a neuron layer.
> > > 2. The output layer is not a hidden layer. (correct your comment in
> > > the code)
>
> > > > > 3. The name of this file is element wise, that is element by elem=
ent =3D
> > every step, Just a precaution to safe guard against any malfunction or =
mist=3D
> > ake in the matrix multiplication done by me. (Although i had created on=
e fi=3D
> > le using matrix multiplication).
>
> > > > > 4. OK i agree i did not know that. Plus i did not know what "loga=
rith=3D
> > mic sigmoid"
>
> > WHOOPS! I meant logistic sigmoid. See below
>
> > > > > is. Are u talking about bipolar sigmoid.
>
> > No. Bipolar sigmoid is a scaled tanh. See beow.
>
> > ( I have not used tanh, I had used bipolar sigmoid.). In the book
> > "Fundamentals ofneuralnetworks by laurene Fausett of pearson
> > education" They had used for both layer the bipolar sigmoid. and
> > according to the text they had converge the weight using the bipolar
> > sigmoid as an activation function on both the layers. (unless i had
> > understood it incorrectly. According to them they had solve the xor
> > problem solution in binary representation, to get converge in 3000
> > epochs.. Mine never did)
>
> > > logsig(x) =3D3D 1/(1+exp(-x))
>
> > Logistic Sigmoid
>
> > > tanh(x/2) =3D3D 2*logsig(x)-1
>
> > > You might be interested in
>
> > >http://groups.google.com/group/comp.ai.neural-nets/
> > > msg/1c33c39b186dd026?hl=3D3Den
>
> > > Note the learning times using the batch L-M algorithm.
>
> > Also
>
> >http://groups.google.com/group/comp.ai.neural-nets/browse_thread/thre...
> > 4b0a03ffcdafb?hl=3D3Den&tvc=3D3D2&q=3D3Dgreg-heath+backpropagation+tuto=
rial+
>
> > Hope this helps.
>
> > Greg
>
> Sir.
> My basic and only concern is "Why did my algo did not converge ?"

You made one or more mistakes. You may have to go through an
iteration
or two by hand.

> Is binary sigmoid is logistic sigmoid ( logsig(x) ) ?

In the last 20 years I have never heard of a binary sigmoid
and I'm glad I haven't because the term does not make sense.
Binary implies two values whereas a sigmoid is real valued.

Did you mean bipolar sigmoid? Bipolar implies it contains
both positive and negative value.

If you have the NN Toolbox type (otherwise read the NN Toolbox
documentation on the MATLAB site)

doc logsig % Logarithmic sigmoid
                      % logsig(n) =3D 1 / (1 + exp(-n))

doc tansig % Hyperbolic tangent sigmoid
                     % tansig(n) =3D 2/(1+exp(-2*n))-1 =3D tanh(n)

Most of the world uses the term logistic sigmoid for logsig.
(Google on both logidtic-sigmoid and logarithmic-sigmoid).
Logsig is unipolar.

Tansig is bipolar.


> You are right. Input layer is not included in the layer count. (which i a=
lso didnot counted).

Be careful. Some authors do count it.

But i
>
> counted output layer because and hidden layer =A0(the layer next to input=
 layer) because both those
>
> layer had weight which did change in the learning process.. Secondly the =
text book also called it
>
> two layer. Another text book "Neural network. Algorithums, applications, =
and programming techniques
>
> by James A. Freeman / David M. Skapura" also called it two layer backprop=
agation network. If i had
>
> understood it incorrectly I would be greatful if u correct me.
> I think in matlab documentation they didnot count the output layer. (i wi=
ll conform that when i
>
> check that) if u r saying according to that then i agree with u.

You missed my point. In your code you called the output layer
the second hidden layer. That is incorrect.

Now that you've mentioned layer count, let me clarify:

Different authors use different ways to count layers.

Some consider layers of nodes
Some consider layers of neurons
Some consider layers of weights

Some count input layers
Some count output layers

I, on the otherhand, am always unambiguous because
I just count hidden layers.


>
> the last link that you sendhttp://groups.google.com/group/comp.ai.neural-=
nets/browse_thread/thre...
>
> c=3D3D2&q=3D3Dgreg-heath+backpropagation+tutorial+
> orhttp://groups.google.com/group/comp.ai.neural-nets/browse_thread/thre..=
.
> didnot render a page the responce was
> Topic not found
> "We're sorry, but we were unable to find the topic you were looking for. =
Perhaps the URL you clicked
>
> on is out of date or broken?

Find the 5 message comp.ai.neural-net thread

Basic Backpropagation Learning Strategy (Tutorial)

The first post is at

http://groups.google.com/group/comp.ai.neural-nets/msg/d35128ccb25c7779?hl=
=3Den

and contains a link to the thread.

Hope this helps.

Greg

Basic Backpropagation Learning Strategy (Tutorial)

http://groups.google.com/group/comp.ai.neural-nets/msg/d35128ccb25c7779?hl=en

Are Linear Output Activations Better than Tanh for XOR?

http://groups.google.com/group/comp.ai.neural-nets/msg/1c33c39b186dd026?hl=en

Hope this helps.

Greg

On Mar 18, 11:46=A0am, Greg Heath <he...@alumni.brown.edu> wrote:
> Basic Backpropagation Learning Strategy (Tutorial)
>
> http://groups.google.com/group/comp.ai.neural-nets/msg/d35128ccb25c77...
>
> Are Linear Output Activations Better than Tanh for XOR?
>
> http://groups.google.com/group/comp.ai.neural-nets/msg/1c33c39b186dd0...

Sorry, the Google newreader is truncating the URLs. The last fields
are

Basic Backpropagation Learning Strategy (Tutorial)

.../msg/d35128ccb25c7779?hl=3Den

Are Linear Output Activations Better than Tanh for XOR?

..//msg/1c33c39b186dd026?hl=3Den

PHEW!

Greg

Sir Thank you for your kind help and time that you had given to me.


I had several time hand executed the epochs, but could not find what wrong with the algorithm. (I had to think really out side the box because my limited mind just could not gasp the core of the problem). And that why I posted the problem on the net.

I quote word by word from the section of (Fausett 1994) book on page 35 so that there are no mistakes done by me.
“Sigmoid functions (S-shaped curves) are useful activation functions. The logistic function and hyperbolic tangent functions are the most common. They are especially advantages for the use in neural net trained by backpropagation, because the simple relation between the values of the function at a point and the values of the derivative at the point reduces the computational burden during raining.
 The logistic function, a sigmoid function with range from 0 to 1, is often used as the activation function for neural nets in which the desired output values either are binary or are in the interval between 0 and 1. to emphasize the range of the function, we call it the binary sigmoid; it is also called the logistic sigmoid.
Binary sigmoid:
                             f(x) = 1/(1 + exp(-x))
logistic sigmoid can be scaled Bipolar to have any range of values that is appropriate for a given problem. The most common range is from -1 to 1; we call this sigmoid the bipolar sigmoid.
                           g(x) = 2 / (1 + exp(-x)) – 1
The hyperbolic tangent is
                           h(x) = (exp(x) – exp(-x)) / (exp(x) + exp(-x))
”

Fausett, L. (1994). Common Activation Functions. Fundamentals of neural networks Architectures, algorithms, and applications, Pearson education (Singapore) Pte. Ltd., Indian Branch, 482 F.I.E. Patparganj, Delhi 110 092, India: 467.


You mention to see MATLAB documentation
doc logsig % Logarithmic sigmoid
                      % logsig(n) =3D 1 / (1 + exp(-n))

Sorry Sir if saw the documentation but never saw where they called it “Logarithmic sigmoid”. They just say
logsig
Log-sigmoid transfer function
Algorithm
logsig(n) = 1 / (1 + exp(-n))
tansig(n) = 2/(1+exp(-2*n))-1
I think, You might be right about the tansig because if algo look very similar to the logsig and a bit to mathematical juggling it might be a multiple of it.

On Mar 19, 1:13=A0pm, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir Thank you for your kind help and time that you had given to me.
>
> I had several time hand executed the epochs, but could not find what wron=
g with the algorithm. (I had to think really out side the box because my li=
mited mind just could not gasp the core of the problem). And that why I pos=
ted the problem on the net.
>
> I quote word by word from the section of =A0(Fausett 1994) book on page 3=
5 so that there are no mistakes done by me.
> “Sigmoid functions (S-shaped curves) are useful activation function=
s. The logistic function and hyperbolic tangent functions are the most comm=
on. They are especially advantages for the use in neural net trained by bac=
kpropagation, because the simple relation between the values of the functio=
n at a point and the values of the derivative at the point reduces the comp=
utational burden during raining.
> =A0The logistic function, a sigmoid function with range from 0 to 1, is o=
ften used as the activation function for neural nets in which the desired o=
utput values either are binary or are in the interval between 0 and 1. to e=
mphasize the range of the function, we call it the binary sigmoid; it is al=
so called the logistic sigmoid.
> Binary sigmoid:
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0f(x) =3D 1/(1 =
+ exp(-x))
> logistic sigmoid can be scaled Bipolar to have any range of values that i=
s appropriate for a given problem. The most common range is from -1 to 1; w=
e call this sigmoid the bipolar sigmoid.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0g(x) =3D 2 / (1 + =
exp(-x)) – 1
> The hyperbolic tangent is
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0h(x) =3D (exp(x) &=
#8211; exp(-x)) / (exp(x) + exp(-x))
> ”
>
> Fausett, L. (1994). Common Activation Functions. Fundamentals of neural n=
etworks Architectures, algorithms, and applications, Pearson education (Sin=
gapore) Pte. Ltd., Indian Branch, 482 F.I.E. Patparganj, Delhi 110 092, Ind=
ia: 467.
>
> You mention to seeMATLABdocumentation
> doc logsig % Logarithmic sigmoid
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 % logsig(n) =3D3D 1 / (1 + ex=
p(-n))
>
> Sorry Sir if saw the documentation but never saw where they called it =
220;Logarithmic sigmoid”. They just say
> logsig
> Log-sigmoid transfer function
> Algorithm
> logsig(n) =3D 1 / (1 + exp(-n))
> tansig(n) =3D 2/(1+exp(-2*n))-1
> I think, You might be right about the tansig because if algo look very si=
milar to the logsig and a bit to mathematical juggling it might be a multip=
le of it.

On Mar 19, 1:13 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir Thank you for your kind help and time that you had
>given to me.

Thank you for the thank you ... but where is the check?

> I had several time hand executed the epochs, but could
not find what wrong with the algorithm. (I had to think
really out side the box because my limited mind just could
not gasp the core of the problem). And that why I posted
>the problem on the net.

What changes have you made to the code that was posted?

Implement randn
Correct the within loop initialization of zin anf yin


> I quote word by word from the section of (Fausett 1994)
book on page 35 so that there are no mistakes done by me.

-----SNIP

>... to emphasize the range of the function, we call it
the binary sigmoid; it is also called the logistic sigmoid.

OK, but Strange that I never heard of it before. Nevertheless,
the term still doesn't make sense because with the same line
of reasoning using bipolar binary coding (i.e., {-1,1}), tanh
could be called a binary sigmoid!

To guage the relative popularity of sigmoid adjectives
I googled the following terms and tabulated the No. of hits
below.

logistic-sigmoid 24,100 ... Use this one
logarithmic-sigmoid 15,300
binary-sigmoid 20,400 ... I'm surprised

bipolar-sigmoid 10,500
unipolar-sigmoid 2,400


-----SNIP

> logistic sigmoid can be scaled Bipolar to have any range
> of values that is appropriate for a given problem.

Scaled AND shifted: Scaling yields (0,2) the shifting then
yields (-1,1).

-----SNIP

f(x) = 1/(1+exp(-x) % logistic (binary,logarithmic)
g(x) = 2*f(x) - 1 = tanh(x/2) % bipolar
h(x) = tanh(x) = g(2*x) = 2*f(2x) - 1 % also known as bipolar

> You mention to see MATLAB documentation
> doc logsig % Logarithmic sigmoid
> % logsig(n) = 1 / (1 + exp(-n))
>
> Sorry Sir I saw the documentation but never saw where they
> called it Logarithmic sigmoid;

Good. Obviously they changed the documentation since my
2004 version.

> I think, You might be right about the tansig because if
algo look very similar to the logsig and a bit to mathematical
>juggling it might be a multiple of it.

Not quite:

logsig = f(x)
tansig = h(x) = 2*f(2x)-1

Therefore therefore logsig is a shifted, scaled AND dilated
version of tansig.

Hope this helps.

Greg

On Mar 19, 5:06=A0pm, Greg Heath <he...@alumni.brown.edu> wrote:
> On Mar 19, 1:13 pm, "Adeel " <neoresearc...@gmail.com> wrote:
>
> > Sir Thank you for your kind help and time that you had
> >given to me.
>
> Thank you for the thank you ... but where is the check?
>
> > I had several time hand executed the epochs, but could
>
> not find what wrong with the algorithm. (I had to think
> really out side the box because my limited mind just could
> not gasp the core of the problem). And that why I posted
>
> >the problem on the net.
>
> What changes have you made to the code that was posted?
>
> Implement randn
> Correct the within loop initialization of zin and yin

Include a maximum epoch limit in the while loop
Change alpha

With my modification of your code i have gotten
solutions in less than 350 epochs (~0.7 sec) on
a 3.2 GHz XP.

Hope this helps.

Greg

It would help tremendously, if you show what exactly changes in my code, to make it work.
You said
Correct the within loop initialization of zin and yin
What do you mean by that? What wrong with the initializations?

I had included a max epoch limit of 600 in the code. But now I had commented this if statement. Because it forces the algorithm to stop before it learned the patterns.

I am not using randn because it returns “Normally distributed random numbers”, where as It is highly recommended to use random values in between (-0.5 to 0.5). Although after yours recommendation. I used randn but still doesn’t got the weight to converge i.e. e to go below 0.05. Max err freezes on 1.

On Mar 20, 9:39 am, "Adeel " <neoresearc...@gmail.com> wrote:

> It would help tremendously, if you show what exactly
> changes in my code, to make it work.

I'm sure it would. However, Since this appears to be school
work, Professor Heath prefers to give you enough hints so
that you can figure it out yourself.

> You said Correct the within loop initialization of zin and yin

zin and yin are never cleared of the values they had for
the previous input.

Initialize with v0 and w0, respectively.


> I had included a max epoch limit of 600 in the code.
> But now I had commented this if statement. Because
> it forces the algorithm to stop before it learned the
> patterns.

Neural net design is a guided trial and error process.
I started with maxepoch =3D 200 and eventually raised it to
3000. Even then some trials didn't reach the 0.05 goal.

There was a distribution of convergence times dependent
on random weight initialization. Nevertheless many runs
converged at epoch < 500.

If you revisit my XOR reference, you'll see that using
the L-M algorithm on the NN Toolbox, maxepoch =3D 100,
MSEgoal =3D 0 and 100 trials, I only got 55-65% (dependent
on output activation) of the nets to converge. Obviously,
it would be interesting to go back, increase maxepoch
significantly, and obtain a histogram of convergence epoch
values.

The bottom line is that some nets will converge to values
above the goal you set. Therefore, you must use a maxpoch
or maxCPU costraint and multiple random initializations.

> I am not using randn because it returns “Normally
> distributed random numbers”

Obviously.

where as It is highly recommended to use random values
in between (-0.5 to 0.5).

Not by Bishop or me. See the 2 message thread

Nonsaturating Initial Weights

http://groups.google.com/group/comp.ai.neural-nets/
browse_thread/thread/0c08b10fc3cd8aeb/
9918dc6b99d0aa46?hl=3Den=E2=9A=BEdc6b99d0aa46

the first post is also at

http://groups.google.com/group/comp.ai.neural-nets/
msg/4c92be6341b4f820?hl=3Den

By the way, what is randc? I don't have it and I can't
find it on the MATLAB website. It looks like it could be
complex with real and imaginary values obtained from rand.

> Although after yours recommendation. I used randn but
> still doesn’t got the weight to converge i.e.
> e to go below 0.05. Max err freezes on 1.

I used logsig in the output and maxerr froze on 0.5
until I changed alpha.

Hope this helps.

Greg

I am sending the pseuodocode of the algorithm with actual code.
randc is
% RANDC returns random matrix with entries in [-0.5,0.5]
% Authors: P. Gahinet and A. Nemirovski 3/95
% Copyright 1995-2004 The MathWorks, Inc.
% $Revision: 1.1.8.1 $
function [M] = randc(m,n)
if nargin==1, n=m; end
M=rand(m,n)-.5*ones(m,n);

function [v,v0,w,w0] = trainBPElementWise(v,v0,w,w0)
%Create an back propagation net
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% sigmak Portion of error correction weight adjustment for wjk that is due
% to an error at the output unit Yk; also, the information about the error
% at unit Yk that is propagated back to the hidden units that feed into
% unit Yk
% sigmaj Portion of error correction weight adjustment for vij that is due
%to the backpropagation of error information from the output layer to the
%hidden unit Zj.
% alpha Learning rate
% Xi Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, xi.
% v0j Bais on the hidden unit j.
% Zj Hidden init j:
% The net input to Zj is denoted z_inj:
% z_inj = v0j + sum(xi*vij).
% The output signal (activation) of Zj is denoted zj:
% zj = f(z_inj).
% w0k Bais on output unit k.
% Yk Output unit k.
% The net input to Yk is denoted y_ink:
% y_ink = w0k + sum(zj*wjk)
% the output signal (activation) of Yk is denoted yk:
% yk = f(y_ink).
% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% (Formula formatted) zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% (Formula formatted) z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% (Formula formatted) Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% (Formula formatted) yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% (Formula formatted) y_k = f(yin_k).
% BackPropagation of error:
% (Formula formatted) Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% (Formula formatted) d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (Formula formatted) (used to update w_(jk) later),
% (Formula formatted) chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (Formula formatted) (used to update w0_k later),
% (Formula formatted) chw0_k = alpha d_k,
% (Formula formatted) and sends d_k to units in the layer
% below.
% (Formula formatted) Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% (Formula formatted) din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% (Formula formatted) d_j =din_j f' (zin_j),
% calculates its weight correction term
% (Formula formatted) (used to update v_(ij) later),
% (Formula formatted) chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (Formula formatted) (used to update v0_j later),
% (Formula formatted) chv0_j = alpha d_j.
% Update weights and baises:
% (Formula formatted) Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% (Formula formatted) w_(jk) (new) = w_(jk) (old) + chw_(jk),
% (Formula formatted) Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% (Formula formatted) v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.

%% ==Step_0==Initialization=================================
clc
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = 2; %INPUT
    p = 4;
    m = 1; % OUTPUT
    alpha = 0.02;
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    t = [0; 1; 1; 0];
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
%{
        v = randc(n,p);
        w = randc(p,m);
        w0 = randc(1,m);
        v0 = rand(1,p);
%}
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;

%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('Epoch : %4g, max err : %d',iteration, errorMax(iteration)));
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = (2/(1+exp(-yin(Tp,k))))-1; %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
    if (~mod(iteration,200))
% figure
        plot(1:200,errorMax(iteration-199:iteration))
        getframe;% F(iteration) = getframe; movie(F);
    end
    iteration = iteration +1;
% if (iteration > 600)
% break
% end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
clf('reset'), cla reset, plot(errorMax)
xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

On Mar 21, 11:28 am, "Adeel " <neoresearc...@gmail.com> wrote:
> I am sending the pseuodocode of the algorithm with actual code.

-----SNIP

Why are you posting this?

I've told you what needs to be changed.

Obviously you haven't made those changes.

In case you've forgotten, I'll list what
I did to make your program run beautifully
(well, except for the weird plot stuff):

% 1. Changed alpha
% 2. Used randn to initialize weights
% 3-5. Initialized zin, yin and din within loops
% 6-7. Changed to Unipolar output & derivative

Puzzled,

Greg

You said
“zin and yin are never cleared of the values they had for the previous input.”
I had cleared all the variables (except the change weights and biases variables i.e. chw, chw0, chv and chv0). I had initialize them in the while loop
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
The error now at least go down than 1 but as soon as it reaches 5.297516e-001 (or 0.5298), it then go up consistently.
You said “I used logsig in the output and maxerr froze on 0.5 until I changed alpha.”
I had changed alpha = 0.00002; but it still does not converged.
You said “Initialize with v0 and w0, respectively.” What does that means?
By the way what does “maxCPU constraint” means? Does Matlab monitor the CPU usage.
 
function [v,v0,w,w0, errorMax] = trainBPElementWise(v,v0,w,w0)
%Create an back propagation net
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% sigmak Portion of error correction weight adjustment for wjk that is due
% to an error at the output unit Yk; also, the information about the error
% at unit Yk that is propagated back to the hidden units that feed into
% unit Yk
% sigmaj Portion of error correction weight adjustment for vij that is due
%to the backpropagation of error information from the output layer to the
%hidden unit Zj.
% alpha Learning rate
% Xi Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, xi.
% v0j Bais on the hidden unit j.
% Zj Hidden init j:
% The net input to Zj is denoted z_inj:
% z_inj = v0j + sum(xi*vij).
% The output signal (activation) of Zj is denoted zj:
% zj = f(z_inj).
% w0k Bais on output unit k.
% Yk Output unit k.
% The net input to Yk is denoted y_ink:
% y_ink = w0k + sum(zj*wjk)
% the output signal (activation) of Yk is denoted yk:
% yk = f(y_ink).
% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
%% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% (Formula formatted) zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% (Formula formatted) z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% (Formula formatted) Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% (Formula formatted) yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% (Formula formatted) y_k = f(yin_k).
%% BackPropagation of error:
% (Formula formatted) Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% (Formula formatted) d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (Formula formatted) (used to update w_(jk) later),
% (Formula formatted) chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (Formula formatted) (used to update w0_k later),
% (Formula formatted) chw0_k = alpha d_k,
% (Formula formatted) and sends d_k to units in the layer
% below.
% (Formula formatted) Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% (Formula formatted) din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% (Formula formatted) d_j =din_j f' (zin_j),
% calculates its weight correction term
% (Formula formatted) (used to update v_(ij) later),
% (Formula formatted) chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (Formula formatted) (used to update v0_j later),
% (Formula formatted) chv0_j = alpha d_j.
%% Update weights and baises:
% (Formula formatted) Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% (Formula formatted) w_(jk) (new) = w_(jk) (old) + chw_(jk),
% (Formula formatted) Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% (Formula formatted) v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.
 
%% ==Step_0==Initialization=================================
clc
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = 2; %INPUT
    p = 4;
    m = 1; % OUTPUT
    alpha = 0.00002;
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    t = [0; 1; 1; 0];
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
%{
        v = randc(n,p);
        w = randc(p,m);
        w0 = randc(1,m);
        v0 = rand(1,p);
%}
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;
 
%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
% errorMax(iteration) = max(max(error));
    disp(sprintf('Epoch : %4g, max err : %d',iteration, max(max(error)))); %errorMax(iteration)));
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = (2/(1+exp(-yin(Tp,k))))-1; %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
% if (~mod(iteration,200))
% % figure
% plot(1:200,errorMax(iteration-199:iteration))
% getframe;% F(iteration) = getframe; movie(F);
% end
    iteration = iteration +1;
% if (iteration > 30000)
% break
% end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
% clf('reset'), cla reset, plot(errorMax)
% xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

>I am sending the pseuodocode of the algorithm with actual code.
 >-----SNIP
 >Why are you posting this?
I send the pseuodocode because you said earlier that I should insert much more comment and pseuodocode is the highest comment that i though

> I've told you what needs to be changed.
>
> Obviously you haven't made those changes.
>
I posted that message before I had made changes to my algorithm.
> In case you've forgotten, I'll list what
> I did to make your program run beautifully
> (well, except for the weird plot stuff):
>
> % 1. Changed alpha
I will include a alpha changing algorithm in the training algo. But first I wanted to get thing at least straight with the training algo. Yeah I know about the alpha matrix and change to individual alphas accordingly. But did not you think that would be a stretch right now.
> % 2. Used randn to initialize weights
> % 3-5. Initialized zin, yin and din within loops
> % 6-7. Changed to Unipolar output & derivative
I actually would use this algo for face recognition, so changing to unipolar would not help be much in the actual problem solved by the algo. (although I would change for now. I will give you the results).
>
> Puzzled,
That hurts.
>
> Greg

I had commented the plot.
And used the Unipolar activation function & derivative for the output layer but at
max error goes as low to 0.5130
Epoch : 1.32064e+006, max err : 5.130456e-001
And seem to freeze there.

here is the code
It freezes at 0.5130
function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)
%Create an back propagation net
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% sigmak Portion of error correction weight adjustment for wjk that is due
% to an error at the output unit Yk; also, the information about the error
% at unit Yk that is propagated back to the hidden units that feed into
% unit Yk
% sigmaj Portion of error correction weight adjustment for vij that is due
%to the backpropagation of error information from the output layer to the
%hidden unit Zj.
% alpha Learning rate
% Xi Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, xi.
% v0j Bais on the hidden unit j.
% Zj Hidden init j:
% The net input to Zj is denoted z_inj:
% z_inj = v0j + sum(xi*vij).
% The output signal (activation) of Zj is denoted zj:
% zj = f(z_inj).
% w0k Bais on output unit k.
% Yk Output unit k.
% The net input to Yk is denoted y_ink:
% y_ink = w0k + sum(zj*wjk)
% the output signal (activation) of Yk is denoted yk:
% yk = f(y_ink).
% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
%% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% (Formula formatted) zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% (Formula formatted) z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% (Formula formatted) Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% (Formula formatted) yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% (Formula formatted) y_k = f(yin_k).
%% BackPropagation of error:
% (Formula formatted) Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% (Formula formatted) d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (Formula formatted) (used to update w_(jk) later),
% (Formula formatted) chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (Formula formatted) (used to update w0_k later),
% (Formula formatted) chw0_k = alpha d_k,
% (Formula formatted) and sends d_k to units in the layer
% below.
% (Formula formatted) Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% (Formula formatted) din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% (Formula formatted) d_j =din_j f' (zin_j),
% calculates its weight correction term
% (Formula formatted) (used to update v_(ij) later),
% (Formula formatted) chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (Formula formatted) (used to update v0_j later),
% (Formula formatted) chv0_j = alpha d_j.
%% Update weights and baises:
% (Formula formatted) Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% (Formula formatted) w_(jk) (new) = w_(jk) (old) + chw_(jk),
% (Formula formatted) Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% (Formula formatted) v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.

%% ==Step_0==Initialization=================================
clc,clear
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = 2; %INPUT
    p = 4;
    m = 1; % OUTPUT
% alpha = 0.00002;
    alpha = input('enter alpha: ');
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    t = [0; 1; 1; 0];
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
%{
        v = randc(n,p);
        w = randc(p,m);
        w0 = randc(1,m);
        v0 = rand(1,p);
%}
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
end
%%%First hidden Layer
% zin = zeros(trainRecords,p);
% z = zeros(trainRecords,p);
% din = zeros(trainRecords,p);
% dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
% yin = zeros(trainRecords,m);
% y = zeros(trainRecords,m);
% d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;

%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
% errorMax(iteration) = max(max(error));
    disp(sprintf('max err : %d', max(max(error)))); % Epoch : %4g, iteration,errorMax(iteration)));
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = 1/(1+exp(-yin(Tp,k))); %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
% if (~mod(iteration,200))
% % figure
% plot(1:200,errorMax(iteration-199:iteration))
% getframe;% F(iteration) = getframe; movie(F);
% end
    iteration = iteration +1;
% if (iteration > 30000)
% break
% end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
% clf('reset'), cla reset, plot(errorMax)
% xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

On Mar 22, 4:17 am, "Adeel " <neoresearc...@gmail.com> wrote:
> You said
> “zin and yin are never cleared of the
> values they had for the previous input.”
> I had cleared all the variables (except the change weights and
> biases variables i.e. chw, chw0, chv and chv0).
>I had initialize them in the while loop
> zin = zeros(trainRecords,p);
> z = zeros(trainRecords,p);
> din = zeros(trainRecords,p);
> dj = zeros(trainRecords,p);
> yin = zeros(trainRecords,m);
> y = zeros(trainRecords,m);
> d = zeros(trainRecords,m);

No.

You have to initialize zin, yin and din for EACH
input.

-----SNIP

> You said “I used logsig in the output
> and maxerr froze on 0.5 until I changed
> alpha.”
> I had changed alpha = 0.00002; but it still
> does not converged.

Have you thought about alpha > 0.02?

> You said “Initialize with v0 and w0,
> respectively.” What does that means?

See below. Understand the difference?

> By the way what does “maxCPU constraint”
> means? Does Matlab monitor the CPU usage.

It will if you want it to. Use doc and help
to investigate the functions

tic, toc, clock, cputime, and etime

However, an upper bound on iterations is sufficient
for this problem

> function [v,v0,w,w0, errorMax] = trainBPElementWise(v,v0,w,w0)

----SNIP

> %%% Enter the Architecture detail
> disp('Enter the Architecture detail');
> n = 2; %INPUT
> p = 4;

p = 2 is a more interesting problem. After all,
since there is no nontraining testing set, all
this net will do is memorize the training set.
(In real world problems memorizing the training
set often degades performance on nontraining
test data). Given that the number of unknown
weights to be estimated is

Nw = (n+1)*p+(P+1)*m = 3*4+5*1 = 18

and the number of training equations to solve is
only

Neq = Ntrn*m = 4 < Nw = 18

there are an infinite number of solutions.

With p = 2, Nw = 3*2+3*1 = 9

which is still greater than Neq = 4, but is more
of a challenge.

> m = 1; % OUTPUT
> alpha = 0.00002;

This choice is KILLING you more than the original
0.02. Think out of the box when searching for a
more suitable value (don't be shy!).

-----SNIP

> %%% Initialization of the weights
> if (nargin < 2)
> disp('weights v and w are getting initialised randomly');
> %{
> v = randc(n,p);
> w = randc(p,m);
> w0 = randc(1,m);
> v0 = rand(1,p);
> %}
> v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
> v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
> 0.3099, 0.1904, -0.0347, -0.4861];
> w0 = -0.1401;
> w = [0.4919; -.2913; -0.3979; 0.3581];

NO!, NO!, NO! Use randn!
-----SNIP

> while er==0
> % errorMax(iteration) = max(max(error));
> disp(sprintf('Epoch : %4g, max err : > %d',iteration, max(max(error))));
> %errorMax(iteration)));

I moved these to the end of the loop

> %%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
> zin = zeros(trainRecords,p);
> z = zeros(trainRecords,p);
> din = zeros(trainRecords,p);
> dj = zeros(trainRecords,p);
> yin = zeros(trainRecords,m);
> y = zeros(trainRecords,m);
> d = zeros(trainRecords,m);

Delete the above statements.

How can this be for EACH training pair when it
is not inside the Tp for loop?

Study the Tp for loop code carefully.

Why do zin, din and yin have to be intialized
inside that loop?.


> for Tp=1:trainRecords
-----SNIP
> %%% First Layer
> for j=1:p
> for i=1:n
> zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
> end
> zin(Tp,j) = v0(j) + zin(Tp,j);

for i=1:n
    zin(Tp,j) = v0(j);
    zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
end

Now do you understand?

> z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
> end
> %%% ==Step_5==================================================
> %%% Output Layer
> for k=1:m
> for j=1:p
> yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
> end
> yin(Tp,k) = w0(k) + yin(Tp,k);

Correct the initialization of yin


> y(Tp,k) = (2/(1+exp(-yin(Tp,k))))-1; %activation function


No. Use the unipolar sigmoid


> end
> %% Backpropagation of error (Training Started)
> %%% ==Step_6==================================================
> for k=1:m
> %%% Error Info = (t - y) * (Derivative of Second layer activation function)
> d(Tp,k) = (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(1 - y(Tp,k)));
> for j=1:p
> chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
> end
> chw0(k) = alpha * d(Tp,k);
> end
> %%% ==Step_7==================================================
> for j=1:p
> for k=1:m
> din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
> end

Initialize din

> %%% Error Info = din * (Derivative of First layer activation function)
> dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
> for i=1:n
> chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
> end
> chv0(j) = alpha * dj(Tp,j);
> end
> %% ==Step_8==Update_weights_and_biases========================
> for k=1:m
> for j=1:p
> w(j,k)=w(j,k)+chw(j,k);
> end
> w0(k)=w0(k)+chw0(k);
> end
> for j=1:p
> for i=1:n
> v(i,j)=v(i,j)+chv(i,j);
> end
> v0(j)=v0(j)+chv0(j);
> end
> end
> %% ==Step_9==Test_stoping_condition==========================
> error = sqrt((t-y).^2);

Same as abs(y-t)

> %%% Update the stopping condition variable
> if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
> er =1;
> else
> er = 0;
> end
> % if (~mod(iteration,200))
> % % figure
> % plot(1:200,errorMax(iteration-199:iteration))
> % getframe;% F(iteration) = getframe; movie(F);
> % end

Good. Keep this plot commented out.

> iteration = iteration +1;
> % if (iteration > 30000)
> % break
> % end
> end %% End of wile loop Step 1
> save('weight6040.dat','v','v0','w','w0')
> % clf('reset'), cla reset, plot(errorMax)
> % xlabel('iteration '), ylabel('error '), title('Plot of the error')
> end %% End of Function / File

This last plot is OK. Could add a red line
at error = 0.05

Hope this helps.

Greg

On Mar 22, 4:32 am, "Adeel " <neoresearc...@gmail.com> wrote:
> >I am sending the pseuodocode of the algorithm with actual code.
>
> >-----SNIP
> >Why are you posting this?
> I send the pseuodocode because you said
>earlier that I should insert much more comment
>and pseuodocode is the highest comment that i
>though

It would have helped 24 replies ago. But now ...

> > I've told you what needs to be changed.
>
> > Obviously you haven't made those changes.
>
> I posted that message before I had made changes to my algorithm.
> In case you've forgotten, I'll list what
> > I did to make your program run beautifully
> > (well, except for the weird plot stuff):
>
> > % 1. Changed alpha
>
> I will include a alpha changing algorithm in
>the training algo. But first I wanted to get
>thing at least straight with the training algo.
>Yeah I know about the alpha matrix and change to
>individual alphas accordingly. But did not you
>think that would be a stretch right now.

That is not what I meant. The value of 0.02 will
not work for me. It is not even close. It must be
changed. If you don't find a suitable value, the
algorithm will not converge in a reasonable amount
of time.

When using trial and error to find a suitable value
think out of the box.

With H = 4 hidden nodes I've gotten convergence
with less than 80 epochs and less than 300 with
H = 2.

> % 2. Used randn to initialize weights
> > % 3-5. Initialized zin, yin and din within loops
> > % 6-7. Changed to Unipolar output & derivative
>
-----SNIP

> > Puzzled,

Because there was no indication in the posted
code that you had tried to make the suggested
changes.

> That hurts.

Don't worry, you'll recover just nicely.

Greg

On Mar 22, 5:04 am, "Adeel " <neoresearc...@gmail.com> wrote:
> I had commented the plot.
> And used the Unipolar activation function & derivative for the output layer but at
> max error goes as low to 0.5130
> Epoch : 1.32064e+006, max err : 5.130456e-001
> And seem to freeze there.

I see from a previous reply that you had tried
reducing alpha.

However, given the other modifications I suggested,
have you thought of "thinking out of the box" and
increasing it?

(;>)

Greg

On Mar 22, 5:30 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> here is the code
> It freezes at 0.5130
> function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)

-----SNIP

> %%% Initialization of the weights
> if (nargin < 2)
> disp('weights v and w are getting initialised randomly');
> %{
> v = randc(n,p);
> w = randc(p,m);
> w0 = randc(1,m);
> v0 = rand(1,p);
> %}
> v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
> v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
> 0.3099, 0.1904, -0.0347, -0.4861];
> w0 = -0.1401;
> w = [0.4919; -.2913; -0.3979; 0.3581];
> end

Change to randn

> %%%First hidden Layer
> % zin = zeros(trainRecords,p);
> % z = zeros(trainRecords,p);
> % din = zeros(trainRecords,p);
> % dj = zeros(trainRecords,p);
> chv = zeros(n,p);
> chv0 = zeros(1,p);
> %%%Output Layer
> % yin = zeros(trainRecords,m);
> % y = zeros(trainRecords,m);
> % d = zeros(trainRecords,m);

Uncomment the above to preallocate storage space

-----SNIP

> while er==0
> % errorMax(iteration) = max(max(error));
> disp(sprintf('max err : %d', max(max(error)))); % Epoch : %4g, iteration,errorMax(iteration)));
> %%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
> zin = zeros(trainRecords,p);
> z = zeros(trainRecords,p);
> din = zeros(trainRecords,p);
> dj = zeros(trainRecords,p);
> yin = zeros(trainRecords,m);
> y = zeros(trainRecords,m);
> d = zeros(trainRecords,m);

delete the above initializations

> for Tp=1:trainRecords
> %% Feed forward:
> %%% ==Step_3==X_get_Input=====Already_done====================
> %%% ==Step_4==================================================
> %%% First Layer
> for j=1:p
> for i=1:n

zin(Tp,j) = v0(j) ; % Initialization

> zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
> end
> zin(Tp,j) = v0(j) + zin(Tp,j);

delete the above

> z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
> end
> %%% ==Step_5==================================================
> %%% Output Layer
> for k=1:m
> for j=1:p

Initialize yin

> yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
> end
> yin(Tp,k) = w0(k) + yin(Tp,k);

delete above

> y(Tp,k) = 1/(1+exp(-yin(Tp,k))); %activation function
> end

-----SNIP

> for j=1:p

Initialize din

> for k=1:m
> din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
> end

-----SNIP

> % if (~mod(iteration,200))
> % % figure
> % plot(1:200,errorMax(iteration-199:iteration))
> % getframe;% F(iteration) = getframe; movie(F);
> % end

If you really want to plot within the loop, add one point each
iteration and use the drawnow command . However, it really
 slows things down.

> iteration = iteration +1;
> % if (iteration > 30000)
> % break
> % end

Uncomment the above lf statement

> end %% End of wile loop Step 1
> save('weight6040.dat','v','v0','w','w0')
> % clf('reset'), cla reset, plot(errorMax)
> % xlabel('iteration '), ylabel('error '), title('Plot of the error')
> end %% End of Function / File

Uncomment the above. This plot is ok.

Why not add a red line at error = 0.05?

Hope this helps.

Greg

Houston, we got a problem.
And that a big one. The algo did even budge an inch (actually one nanometer from max err = 1).
I tried (as you said) changing alpha to 0.04 and then 0.06 etc. Actually now even decreasing the alpha to 0.0002 or 0.00002 etc does not had any effect on the algorithm.

I seriously does not understand the logic behind adding bais to every weight * input (or hidden sum). Because in any neural network, bais is added once and not to all the w*x.
You are changing the whole structure of the neural network. (or you yanking my chin i.e. testing me how stupid I am).
Here is the code

function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)
%Create an back propagation net
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% sigmak Portion of error correction weight adjustment for wjk that is due
% to an error at the output unit Yk; also, the information about the error
% at unit Yk that is propagated back to the hidden units that feed into
% unit Yk
% sigmaj Portion of error correction weight adjustment for vij that is due
%to the backpropagation of error information from the output layer to the
%hidden unit Zj.
% alpha Learning rate
% Xi Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, xi.
% v0j Bais on the hidden unit j.
% Zj Hidden init j:
% The net input to Zj is denoted z_inj:
% z_inj = v0j + sum(xi*vij).
% The output signal (activation) of Zj is denoted zj:
% zj = f(z_inj).
% w0k Bais on output unit k.
% Yk Output unit k.
% The net input to Yk is denoted y_ink:
% y_ink = w0k + sum(zj*wjk)
% the output signal (activation) of Yk is denoted yk:
% yk = f(y_ink).
% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
%% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% (Formula formatted) zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% (Formula formatted) z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% (Formula formatted) Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% (Formula formatted) yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% (Formula formatted) y_k = f(yin_k).
%% BackPropagation of error:
% (Formula formatted) Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% (Formula formatted) d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (Formula formatted) (used to update w_(jk) later),
% (Formula formatted) chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (Formula formatted) (used to update w0_k later),
% (Formula formatted) chw0_k = alpha d_k,
% (Formula formatted) and sends d_k to units in the layer
% below.
% (Formula formatted) Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% (Formula formatted) din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% (Formula formatted) d_j =din_j f' (zin_j),
% calculates its weight correction term
% (Formula formatted) (used to update v_(ij) later),
% (Formula formatted) chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (Formula formatted) (used to update v0_j later),
% (Formula formatted) chv0_j = alpha d_j.
%% Update weights and baises:
% (Formula formatted) Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% (Formula formatted) w_(jk) (new) = w_(jk) (old) + chw_(jk),
% (Formula formatted) Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% (Formula formatted) v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.
 
%% ==Step_0==Initialization=================================
clc,clear
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = 2; %INPUT
    p = 2;
    m = 1; % OUTPUT
% alpha = 0.00002;
    alpha = input('enter alpha: ');
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    t = [0; 1; 1; 0];
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randn(n,p);
        w = randn(p,m);
        w0 = randn(1,m);
        v0 = randn(1,p);
%{
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
%}
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;
 
%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('max err : %d', errorMax(iteration))); % Epoch : %4g, iteration,
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            for i=1:n
                zin(Tp,j) = v0(j);
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
% zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            for j=1:p
                yin(Tp,k) = w0(k);
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
% yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = 1/(1+exp(-yin(Tp,k))); %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
% if (~mod(iteration,200))
% % figure
% plot(1:200,errorMax(iteration-199:iteration))
% getframe;% F(iteration) = getframe; movie(F);
% end
    iteration = iteration +1;
    if (iteration > 30000)
        break
    end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
erLine = ones(1,size(errorMax))*0.05;
clf('reset'), cla reset, plot(1:size(errorMax,2),errorMax,erLine,'r')
xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

On Mar 23, 6:17=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> Houston, we got a problem.
> And that a big one. The algo did even budge an inch (actually one nanomet=
er from max err =3D 1).
> I tried (as you said) changing alpha to 0.04 and then 0.06 etc.

Changing 0.02 to 0.06 is not thinking out of the box.
What, precisely is "etc" supposed to mean?

>Actually now even decreasing the alpha to 0.0002 or 0.00002 etc
>does not had any effect on the algorithm.

Why doesn't it occur to you that you are going the wrong way?

Why are you decreasing it even further after I said alpha > 0.02???

> I seriously does not understand the logic behind adding bais to every wei=
ght * input (or hidden sum). Because in anyneuralnetwork, bais is added onc=
e and not to all the w*x.
> You are changing the whole structure of the neuralnetwork.

No. You are just not coding the matrix multiplications correctly.
Why didn't you use the syntax in my hint?

> (or you yanking my chin i.e. testing me how stupid I am).

I am serious. You need practice with nested loops.

> Here is the code

Obviously you didn't run this before posting:

1. The code BOMBS because BOTH statements
erLine =3D... and clf... are incorrect
2. You are not serious about using alpha > 0.02
3 You are still not using randn
4. zin, yin and din are still incorrectly initialized.
   a. Delete the initializations at the beginning of the Tp for loop.
   b. Why don't you use the exact syntax I suggested in the inner
loops???
5. Don't expect convergence to a correct answer for every random
weight initialization (remember the results of my linear, tanh and
softmax XOR theads re the % of times the algorithm converged
to a correct answer?

Hope this helps.

Greg


> %% =A0=3D=3DStep_0=3D=3DInitialization=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> clc,clear
> %%% Enter the Architecture detail
> =A0 =A0 disp('Enter the Architecture detail');
> =A0 =A0 n =3D 2; =A0%INPUT
> =A0 =A0 p =3D 2;
> =A0 =A0 m =3D 1; % OUTPUT
> % =A0 =A0 alpha =3D 0.00002;

NOT EVEN CLOSE (neither is 0.02!)

> =A0 =A0 alpha =3D input('enter alpha: ');
> =A0 =A0 e =3D 0.05; % error limit
> % =A0-----------------------------------------------------------
> %%% Initialization of the Input / Target
> =A0 =A0 disp('Loading the input vector x and target vectors')
> =A0 =A0 x =3D [0 0; 0 1; 1 0; 1 1];
> =A0 =A0 t =3D [0; 1; 1; 0];
> =A0 =A0 trainRecords =3D size(x,1);
> % =A0----------------------------------------------------------
> %%% Initialization of the weights
> if (nargin =A0< 2)
> =A0 =A0 disp('weights v and w are getting initialised randomly');
> =A0 =A0 =A0 =A0 v =3D randn(n,p);
> =A0 =A0 =A0 =A0 w =A0=3D randn(p,m);
> =A0 =A0 =A0 =A0 w0 =3D randn(1,m);
> =A0 =A0 =A0 =A0 v0 =3D randn(1,p); =A0 =A0 =A0 =A0
> %{ =A0 =A0
> =A0 =A0 v0 =3D [-0.3378, 0.2771, =A00.2859, -0.3329]; =A0 =A0 =A0
> =A0 =A0 v =A0=3D [ 0.1970, 0.3191, -0.1448, =A00.3594; ... =A0 =A0
> =A0 =A0 =A0 =A0 =A0 =A00.3099, 0.1904, -0.0347, -0.4861];
> =A0 =A0 w0 =3D =A0-0.1401; =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0
> =A0 =A0 w =A0=3D [0.4919; -.2913; -0.3979; 0.3581];
> %}

Why are you still using these ????
 =A0 =A0 =A0 =A0
> end
> %%%First hidden Layer
> =A0 =A0 zin =3D zeros(trainRecords,p);
> =A0 =A0 z =3D =A0zeros(trainRecords,p);
> =A0 =A0 din =3D zeros(trainRecords,p);
> =A0 =A0 dj =3D =A0zeros(trainRecords,p);
> =A0 =A0 chv =3D zeros(n,p);
> =A0 =A0 chv0 =3D zeros(1,p);
> %%%Output Layer
> =A0 =A0 yin =3D zeros(trainRecords,m);
> =A0 =A0 y =3D zeros(trainRecords,m);
> =A0 =A0 d =3D zeros(trainRecords,m);
> =A0 =A0 chw =3D zeros(p,m);
> =A0 =A0 chw0 =3D zeros(1,m);
> iteration =3D1;
> er =3D 0; error =3D 0;
>
> %% =A0=3D=3DStep_1=3D=3DWhile_stoping_condition_is_false=3D=3Ddo_step_2-9=
=3D=3D=3D
> while er=3D=3D0
> =A0 =A0 errorMax(iteration) =3D max(max(error));
> =A0 =A0 disp(sprintf('max err : %d', errorMax(iteration))); % Epoch : %4g=
, iteration,
> %%% =3D=3DStep_2=3D=3DFor_Each_Training_pair=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3Ddo_Steps_3-8=3D=3D=3D
> =A0 =A0 for Tp=3D1:trainRecords
> %% Feed forward: =A0 =A0 =A0
> %%% =3D=3DStep_3=3D=3DX_get_Input=3D=3D=3D=3D=3DAlready_done=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =A0 =A0 zin =3D zeros(trainRecords,p);
> =A0 =A0 z =3D =A0zeros(trainRecords,p);
> =A0 =A0 din =3D zeros(trainRecords,p);
> =A0 =A0 dj =3D =A0zeros(trainRecords,p);
> =A0 =A0 yin =3D zeros(trainRecords,m);
> =A0 =A0 y =3D zeros(trainRecords,m);
> =A0 =A0 d =3D zeros(trainRecords,m);

Delete these!

> %%% =3D=3DStep_4=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> %%% First Layer
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zin(Tp,j) =3D v0(j);

Do you really believe that initialization statement belongs there?

> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zin(Tp,j) =3D zin(Tp,j) + x(Tp,i) * v(i,j=
);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> % =A0 =A0 =A0 =A0 =A0 =A0 zin(Tp,j) =3D v0(j) + zin(Tp,j);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0z(Tp,j) =3D (2/(1+exp(-zin(Tp,j))))-1; %activa=
tion function
> =A0 =A0 =A0 =A0 end
> %%% =3D=3DStep_5=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 %%% Output Layer
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 yin(Tp,k) =3D w0(k);

BZZT!

> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 yin(Tp,k) =3D yin(Tp,k) + z(Tp,j) * w(j,k=
);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> % =A0 =A0 =A0 =A0 =A0 =A0 yin(Tp,k) =3D w0(k) + yin(Tp,k);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0y(Tp,k) =3D 1/(1+exp(-yin(Tp,k))); %activation=
 function
> =A0 =A0 =A0 =A0 end
> %% Backpropagation of error (Training Started)
> %%% =3D=3DStep_6=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 %%% Error Info =3D (t - y) * (Derivative of Second layer activati=
on function)
> =A0 =A0 =A0 =A0 d(Tp,k) =3D (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k))=
);
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 chw(j,k) =3D alpha * d(Tp,k) * z(Tp,j);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 chw0(k) =3D alpha * d(Tp,k);
> =A0 =A0 =A0 =A0 end
> %%% =3D=3DStep_7=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 din(Tp,j) =3D din(Tp,j) + d(Tp,k) * w(j,k=
);
> =A0 =A0 =A0 =A0 =A0 =A0 end

din needs to be initialized

> %%% =A0 =A0 Error Info =3D din * (Derivative of First layer activation fu=
nction)
> =A0 =A0 =A0 =A0 =A0 =A0dj(Tp,j) =3D (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* =
(1 - z(Tp,j))));
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 chv(i,j) =3D alpha * dj(Tp,j) * x(Tp,i);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0 chv0(j) =3D alpha * dj(Tp,j);
> =A0 =A0 =A0 =A0 end
> %% =3D=3DStep_8=3D=3DUpdate_weights_and_biases=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =A0 =A0 =A0 =A0 for k=3D1:m
> =A0 =A0 =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 w(j,k)=3Dw(j,k)+chw(j,k);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0w0(k)=3Dw0(k)+chw0(k);
> =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 for j=3D1:p
> =A0 =A0 =A0 =A0 =A0 =A0 for i=3D1:n
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 v(i,j)=3Dv(i,j)+chv(i,j);
> =A0 =A0 =A0 =A0 =A0 =A0 end
> =A0 =A0 =A0 =A0 =A0 =A0v0(j)=3Dv0(j)+chv0(j);
> =A0 =A0 =A0 =A0 end
> =A0 =A0 end
> %% =A0=3D=3DStep_9=3D=3DTest_stoping_condition=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =A0 =A0 error =3D sqrt((t-y).^2);
> =A0 =A0 %%% Update the stopping condition variable
> =A0 =A0 if max(max(error)) < e % W/P =3D e, W =3Dweights,P =3Dinputs, e =
=3D errors
> =A0 =A0 =A0 =A0 er =3D1;
> =A0 =A0 else
> =A0 =A0 =A0 =A0 er =3D 0;
> =A0 =A0 end
> % =A0 =A0 if (~mod(iteration,200))
> % % =A0 =A0 =A0 =A0 figure
> % =A0 =A0 =A0 =A0 plot(1:200,errorMax(iteration-199:iteration))
> % =A0 =A0 =A0 =A0 getframe;% =A0 =A0 F(iteration) =3D getframe; movie(F);
> % =A0 =A0 end
> =A0 =A0 iteration =3D iteration +1;
> =A0 =A0 if (iteration > 30000)

3000 is more than enough

> =A0 =A0 =A0 =A0 break
> =A0 =A0 end
> end %% End of wile loop Step 1
> save('weight6040.dat','v','v0','w','w0')
> erLine =3D ones(1,size(errorMax))*0.05;

Incorrect

> clf('reset'), cla reset, plot(1:size(errorMax,2),errorMax,erLine,'r')

Incorrect

> xlabel('iteration '), ylabel('error '), title('Plot of the error')
> end %% End of Function / File

Obviously I did not even reach the erLine and plot statements because when I saw that max error is not budging there is no point in going forward.
(I just stopped it through Ctrl + c).

>Changing 0.02 to 0.06 is not thinking out of the box.
>What, precisely is "etc" supposed to mean?

Well I don’t know I just incremented it on the next trial. Is there a formula for calculating the precise alpha. (Well if there is I surely doesn’t know of it).

>Why doesn't it occur to you that you are going the wrong way?
Ok how would I judge that I am going the wrong way. (How would you decide when in both cases (increasing or decreasing alpha) doesn’t make any difference. i.e. doesn’t make it converge.

>No. You are just not coding the matrix multiplications correctly.
>Why didn't you use the syntax in my hint?
Well So far what I know is that
bais is added to the sum(input * weight).
And you are suggesting adding
Sum(input * weight + bais)

>You are still not using randn
Well I am using it. The %{ token is used as an block comment in Matlab. So all the statements after it are not parsed by the interpreter . until the %} token appears. So you see all the statement that you think that I am
%{
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
%}
are actually commented.
>Why are you still using these ????
Sir o Sir o Sir I am not using them. (I swear).

I did not expects convergence of every random number but I did expect it to converge for at least once in a blue moon (other wise what the point).
Earlier you said:
>> for i=1:n
>zin(Tp,j) = v0(j) ; % Initialization
>> zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
> >end
>> zin(Tp,j) = v0(j) + zin(Tp,j);
>delete the above
Now:
>4. zin, yin and din are still incorrectly initialized.
> a. Delete the initializations at the beginning of the Tp for loop.
> b. Why don't you use the exact syntax I suggested in the inner
>loops???
>Do you really believe that initialization statement belongs there?
No I did not think that the correct form that why I was telling you that bais are never added to the x * w. But you suggested, so I did it.

I think you are suggesting that, instead of initialization at the top of the training loop, I should initialize the matrices in the loop. i.e. din should be initialize as you said in step 7.
When I just summed it.? (before using it).
I think you are saying that I should reinialize it after I calculated the d. (but what difference it made either I initialize it in the beginning of the loop or in the loop?).

These are the correct statements.

erLine = ones(1,size(errorMax,2))*0.05;
plot(1:size(errorMax,2),errorMax,1:size(errorMax,2),erLine,'r')

>Changing 0.02 to 0.06 is not thinking out of the box.
 Do you mean 0.2 or 0.3 etc? (Etc means increment in that way).

>Why don't you use the exact syntax I suggested in the inner loops???
Because I did not understand you mean. First you said do insert the adding of bias, While the summation of x * w is being done. Then you said “Do you really believe that initialization statement belongs there?”. Or had you suggested something else that I had missed.

I am now using input() function to give dynamically give alpha value instead of hard coding it. As you suggested to try, try again to find the optimal value of alpha.

On Mar 24, 3:35 am, "Adeel " <neoresearc...@gmail.com> wrote:
> Obviously I did not even reach the erLine and plot
> statements because when I saw that max error is not
> budging there is no point in going forward.
> (I just stopped it through Ctrl + c).

If you had put in a break when iteration > 1000 (or even 3000
as I had originally suggested), you wouldn't have to use control
C and the rest of the program could have been debugged before
posting.

> >Changing 0.02 to 0.06 is not thinking out of the box.
> >What, precisely is "etc" supposed to mean?
>
> Well I don’t know I just incremented it on the
> next trial. Is there a formula for calculating the
> precise alpha. (Well if there is I surely doesn’t
> know of it).

No there isn't. I just used trial and error. However,
after failing for alpha <= 0.02, I thought out of
the box and investigated 0.02 < alpha < 4.0

> >Why doesn't it occur to you that you are going the wrong way?
>
> Ok how would I judge that I am going the wrong way. (How would
> you decide when in both cases (increasing or decreasing alpha)
> doesn’t make any difference. i.e. doesn’t make it
> converge.

Like you, I originally decreased alpha until it became
clear that all that I did was to delay the results for the
higher values that I tried. Then I "thought out of the box"
and increased it.

> >No. You are just not coding the matrix multiplications
> > correctly.
> >Why didn't you use the syntax in my hint?
>
> Well So far what I know is that
> bais is added to the sum(input * weight).

Yes

> And you are suggesting adding
> Sum(input * weight + bais)

No. Take a closer look :

        for j=1:p
            zin(Tp,j) = v0(j);
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1;
        end

is the same as

        for j=1:p
            zin(Tp,j) = 0;
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
            z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1;
        end

> >You are still not using randn
>
> Well I am using it. The %{ token is used as an block comment
> in Matlab. So all the statements after it are not parsed by
> the interpreter . until the %} token appears. So you see all
> the statement that you think that I am
> %{
> v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
> v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
> 0.3099, 0.1904, -0.0347, -0.4861];
> w0 = -0.1401;
> w = [0.4919; -.2913; -0.3979; 0.3581];
> %}
> are actually commented.
>
> >Why are you still using these ????
>
> Sir o Sir o Sir I am not using them. (I swear).

Hmmm. That feature must have been put in after my version
(6.5.1). I have to comment out each line separately.

Sorry for the scolding. However, just to be absolutely sure,
try removing the semicolons and adding the command

v0,v,w0,w

also without a semicolon.

> I did not expects convergence of every random number but I did
> expect it to converge for at least once in a blue moon (other
> wise what the point).

I obtained very interesting results when I ran 100 trials each
for alpha = 0.1:0.1:4.0 (Yep, 4000 trials in 5.7 min) and plotted
the min, median, mean and max of maxerror vs alpha. The mean over
100 trials never reached the goal of 0.05. However, the min and
median did. Typically ~0.55 < max < ~ 0.8 except for alpha = 0.8
when max = 1.

I also obtained interesting results by plotting the number of
epochs vs alpha. I included epoch <= 1000 in the while staement.
(min,median,mean) = (1000,1000,1000) until alpha = (0.7,0.8,1.1)
then decreased to (196,305,513) at alpha = 4.

When I get time I will calculate the % of successful convergences
vs alpha.

I won't be able to repeat the results exactly because I didn't
record the initial state of the RNG.

I think the reason that alpha is so high is that you have factors of
1/2 in the weight update formulae. Hoqwever, since you are trying
to differentiate (t-y)^2, the factor should be 2. Therefore, your
weight updates for w are a factor of 4 too small and the weight
updates for v are a factor of 16 too small.

> Earlier you said:>> for i=1:n
> >zin(Tp,j) = v0(j) ; % Initialization
> >> zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
> > >end
> >> zin(Tp,j) = v0(j) + zin(Tp,j);
> >delete the above
> Now:
> >4. zin, yin and din are still incorrectly initialized.
> > a. Delete the initializations at the beginning of the Tp for loop.
> > b. Why don't you use the exact syntax I suggested in the inner
> >loops???
> >Do you really believe that initialization statement belongs there?
>
> No I did not think that the correct form that why I was telling you that bais are never added to the x * w. But you suggested, so I did it.
>
> I think you are suggesting that, instead of initialization at the top of the training loop, I should initialize the matrices in the loop. i.e. din should be initialize as you said in step 7.
> When I just summed it.? (before using it).
> I think you are saying that I should reinialize it after I calculated the d. (but what difference it made either I initialize it in the beginning of the loop or in the loop?).

See above.

Hope this helps.

Greg

On Mar 24, 10:22 am, Greg Heath <he...@alumni.brown.edu> wrote:
-----SNIP

> I obtained very interesting results when I ran 100 trials each
> for alpha = 0.1:0.1:4.0 (Yep, 4000 trials in 5.7 min) and plotted
> the min, median, mean and max of maxerror vs alpha.

The curves for the min and median were ,basically, monotonically
decreasing until the goal of 0.05. The curves for the mean and
max fluctuated (significantly for the latter). Trends are descibed
below:

The min decreased from 0.373 @ alpha = 0.1 to the goal of
0.05 @ alpha = 0.8 and beyond.
The median was 0.542 @ alpha = 0.1, increased to 0.560 @
alpha = 0.2 , then decreased to the goal of 0.05 @ alpha = 1.2
and beyond.

> The mean over 100 trials never reached the goal of 0.05.

It decreased from 0.553@ alpha = 0.1 to 0.135 @alpha = 1.2
then increased to 0.213 at alpha = 4.

Typically ~0.55 < max < ~ 0.8 except for a peak of max = 1
@alpha = 0.8

> I also obtained interesting results by plotting the number of
> epochs vs alpha. I included epoch <= 1000 in the while staement.
> (min,median,mean) = (1000,1000,1000) until alpha = (0.7,0.8,1.1)
> then decreased to (196,305,513) at alpha = 4.
>
> When I get time I will calculate the % of successful convergences
> vs alpha.

pctconv = 0% for alpha < 0.7, increases to 74% @ alpha
= 1.6, then fluctuates moderately for alpha > 1.6 with a local
max of 81% @alpha = 1.8, a local min of 56% @alpha = 3.7,
and ending at 68% for alpha = 4.

Therefore, alphaopt ~ 1.8.

HOWEVER, when the weight updating factors of 1/2 are replaced
by the correct factors of 2, these results could be duplicated
only by using a different alpha for each layer AND knowing the
initial state of the RNG.

For future use

1. Record the intial RNG state
2. Change the factor of 1/2 to 2 in the updating formulae.
3. Use lower values of alpha (in the range 0.1 to 0.5???)

Hope this helps.

Greg

Sir you are great.
I finally made it. I had pasted the final version of the element wise Backpropagation neural network code.
Of course now I will convert it into matrix form. (Because until now I had been using almost any Matlab feature. This could well be converted into C++ (or written in ).

According to the textbooks, Alpha range is from 0 to 1.
So using alpha range from 0.02 to 4, doesn’t exists for me. Yet alone could be reached by me. That seriously thinking out side the box.

You could also block comment by select a group of sentences through mouse of keyboard (shift + up ) and then press ctrl + r (to uncomment press ctrl + t ).

Here is the final code
function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)
%Create an back propagation net
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% sigmak Portion of error correction weight adjustment for wjk that is due
% to an error at the output unit Yk; also, the information about the error
% at unit Yk that is propagated back to the hidden units that feed into
% unit Yk
% sigmaj Portion of error correction weight adjustment for vij that is due
%to the backpropagation of error information from the output layer to the
%hidden unit Zj.
% alpha Learning rate
% Xi Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, xi.
% v0j Bais on the hidden unit j.
% Zj Hidden init j:
% The net input to Zj is denoted z_inj:
% z_inj = v0j + sum(xi*vij).
% The output signal (activation) of Zj is denoted zj:
% zj = f(z_inj).
% w0k Bais on output unit k.
% Yk Output unit k.
% The net input to Yk is denoted y_ink:
% y_ink = w0k + sum(zj*wjk)
% the output signal (activation) of Yk is denoted yk:
% yk = f(y_ink).
% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
%% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% (Formula formatted) zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% (Formula formatted) z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% (Formula formatted) Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% (Formula formatted) yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% (Formula formatted) y_k = f(yin_k).
%% BackPropagation of error:
% (Formula formatted) Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% (Formula formatted) d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (Formula formatted) (used to update w_(jk) later),
% (Formula formatted) chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (Formula formatted) (used to update w0_k later),
% (Formula formatted) chw0_k = alpha d_k,
% (Formula formatted) and sends d_k to units in the layer
% below.
% (Formula formatted) Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% (Formula formatted) din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% (Formula formatted) d_j =din_j f' (zin_j),
% calculates its weight correction term
% (Formula formatted) (used to update v_(ij) later),
% (Formula formatted) chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (Formula formatted) (used to update v0_j later),
% (Formula formatted) chv0_j = alpha d_j.
%% Update weights and baises:
% (Formula formatted) Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% (Formula formatted) w_(jk) (new) = w_(jk) (old) + chw_(jk),
% (Formula formatted) Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% (Formula formatted) v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.

%% ==Step_0==Initialization=================================
clc,clear
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = 2; %INPUT
    p = 2;
    m = 1; % OUTPUT
% alpha = 0.00002;
    alpha = input('enter alpha: ');
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
    x = [0 0; 0 1; 1 0; 1 1];
    t = [0; 1; 1; 0];
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randn(n,p);
        w = randn(p,m);
        w0 = randn(1,m);
        v0 = randn(1,p);
%{
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
%}
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0;

%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('max err : %d', errorMax(iteration))); % Epoch : %4g, iteration,
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            zin(Tp,j) = 0;
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            yin(Tp,k) = 0;
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = 1/(1+exp(-yin(Tp,k))); %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
        d(Tp,k) = 0;
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            din(Tp,j) = 0;
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
            dj(Tp,j) = 0;
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
% if (~mod(iteration,200))
% % figure
% plot(1:200,errorMax(iteration-199:iteration))
% getframe;% F(iteration) = getframe; movie(F);
% end
    iteration = iteration +1;
    if (iteration > 3000)
        break
    end
end %% End of wile loop Step 1
save('weight6040.dat','v','v0','w','w0')
erLine = ones(1,size(errorMax,2))*0.05;
clf('reset'), cla reset
plot(1:size(errorMax,2),errorMax,1:size(errorMax,2),erLine,'r')
xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

When I started coding for the first time I made a blunder as you will see in the code that I had pasted below. I did not know in the beginning that I had made a blunder but the funny thing is that the algorithm did converged. Later I known that the convergence had nothing to do with the inputs or weights. It did converged solely based ibnthe biases alone. For a while I was think (when the real code i.e. with out blunder did not converged ) that I should summit that instead. Who would know. Because there are no teacher in our university that are specialized in neural network.
But the thing is that as the saying goes
“if you are not satisfied with yours work no body else would be” Not to say how terrified you are that somebody would get know of the fact.

Here is the code.
function [v,v0,w,w0] = trainBackPropagationNetAll(v,v0,w,w0)
%% ==Step_0==Initialization=================================
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    TotalRecord = 1520; %No of Records
    e = 0.05; % error limit
    n = 40; % INPUT
    p = 2; %ceil(TotalRecord*e); % HIDDEN Because W/p = e and 100% is 76
    m = 5; % OUTPUT
    alpha = input('Enter the Learning rate : ');
    alp = input('Enter the Momentum rate : ');
    beta = 0.7 * p^(1/n);
    %mov = struct('cdata',{},'colormap',{});
% -----------------------------------------------------------
%%% Opening of the Input / Target Files
    cd(getenv('thesis'))
    paths
    disp('Loading the input vector x and target vectors')
    cd(strcat(getenv('thesis'),'\res\database\Face Database\ExtraFeatures\points_20'));
    x = zeros(TotalRecord,n); x = single(x);
    t1 = false(TotalRecord,m);
    for var=1:TotalRecord %1520
        [x1(var,:),t1(var,:)] = getFileData(sprintf('bioid_%04d.pts', var)); %row wise
    end
    r = randperm(1520);
    for i=1:1520
        x(i,:) = x1(r(i),:);
        tA(i,:) = t1(r(i),:);
    end
    x1 = x;
    [x(1,:),ps] = mapminmax(double(x1(1,:)));
    for o =2:TotalRecord
        x(o,:) = mapminmax('apply',double(x1(o,:)),ps);
    end
    x = single(x);
    [t,TS] = mapminmax(double(tA),-.9,.9); % tV = mapminmax('reverse',t,TS);
    t = single(t);
    clear r tA thesis train var x1 t1
    % disp('Input vector') ,disp(x1)
    % disp('Target vector'), disp(t1)
% ----------------------------------------------------------
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randc(n,p);
        w = randc(p,m);
        w0 = randc(TotalRecord,m);
        vj = sqrt(sum(v.^2));
        for j =1:p
            v(:,j) = (beta * v(:,j)) ./ vj(j);
        end
        v0 = rand(TotalRecord,p)-beta*ones(TotalRecord,p);
end
%%%First hidden Layer
    zin = zeros(TotalRecord,p);
    z = zeros(TotalRecord,p);
    dinj = zeros(TotalRecord,p);
    dj = zeros(TotalRecord,p);
    chv = zeros(n,p);
    chv0 = zeros(TotalRecord,p);
%%%Second hidden Layer
    yin = zeros(TotalRecord,m);
    y = zeros(TotalRecord,m);
    dk = zeros(TotalRecord,m);
    chw = zeros(p,m);
    chw0 = zeros(TotalRecord,m);
iteration =1;
er = 0; error = 0;

%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('Epoch: %4d, max err: %d ',iteration, errorMax(iteration)));
% totalerr = 0;
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
%% Feed forward:
%%% ==Step_3==X_get_Input=====================================
%%% ==Step_4==================================================
%%% First Layer
    zin = x * v + v0;
    z = ((2./(1+exp(-zin)))-1); %activation function
% z = (exp(zin)-exp(-zin))./(exp(zin)+exp(-zin)); %activation function
% z = tansig(zin);
%%% ==Step_5==================================================
%%% Second Layer
    yin = z * w + w0;
% y = tansig(yin);
    y = ((2./(1+exp(-yin)))-1); %activation function
% y = (exp(yin)-exp(-yin))./(exp(yin)+exp(-yin)); %activation function
% totalerr = 0.5 * ((t(T,j) - y(T,j))^2)+totalerr;
%% Backpropagation Training Started
%%% ==Step_6==================================================
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
    dk = (t - y).*((1/2).*(1+y).*(1 - y));
% dk = (t - y) .* ((1+y) .*(1 - y));
% dk = tansig('dn',y);
%%% ==Step_7==================================================
    for T=1:TotalRecord
        for j =1:p
            if(j == 1)
                chw(j,:) = alpha * dk(T,:) * z(T,j);
                dinj(T,j) = sum(dk(T,:) .* w(j,:));
%%% Error Info = dinj * (Derivative of First layer activation function)
                dj(T,j) = (dinj(T,j) * ((1/2) * (1 + z(T,j))* (1 - z(T,j))));
% dj(T,j) = (dinj(T,j) * ((1 + z(T,j))* (1 - z(T,j))));
    % dj = tansig('dn',z);
                chv(:,j) = alpha * dj(T,j) * x(T,:);
                continue
            end
            chw(j,:) = alpha * dk(T,:) * z(T,j) + alp * (chw(j,:)- chw(j-1,:));
            dinj(T,j) = sum(dk(T,:) .* w(j,:));
%%% Error Info = dinj * (Derivative of First layer activation function)
            dj(T,j) = (dinj(T,j) * ((1/2) * (1 + z(T,j))* (1 - z(T,j))));
% dj(T,j) = (dinj(T,j) * ((1 + z(T,j))* (1 - z(T,j))));
% dj = tansig('dn',z);
            chv(:,j) = alpha * dj(T,j) * x(T,:) + (alp * (chv(:,j) -chv(:,j-1)))';
% chv(j,:) = alpha * dj(T,:) * x(T,j);
        end
% w = w + chw;
% v = v + chv;
    end
    chw0 = alpha * dk;
    chv0 = alpha * dj;
%% ==Step_8==Update_weights_and_biases========================
    v = v + chv;
    v0 = v0 + chv0;
    w = w + chw;
    w0 = w0 + chw0;
    % h1 = subplot(2,1,1); surf(h1,im2double(v));
    % h2 = subplot(2,1,2); surf(h2,im2double(w));
    % getframe;% F(iteration) = getframe; movie(F);
    %disp('value of y at this iteration');
    %disp(y)
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
    iteration = iteration +1;
    % alpha = alpha * .999;
    %finerr = totalerr/(TotalRecord*n);
    %disp(finerr)
    %if finerr < 0.01 er = 1; else er = 0; end
    %disp(sprintf('the error : %d, finerr : %d',error , finerr));
    % img = error *255;
    % img = padarray(img, [0 14]);
    % mov(iteration).cdata = double(img);
    % mov(iteration).colormap = [];
    %codecs = videoWriter([],'codecs')
    % if (iteration >=1500)
    % return;
    % end
end %% End of while loop Step 1
% movie2avi(mov, 'MatMov.avi', 'compression', 'Indeo5');
save('weightAll.dat','v','v0','w','w0');
clf('reset'), cla reset, plot(errorMax)
xlabel('iteration '), ylabel('error '), title('Plot of the error')
end %% End of Function / File

Sir Greg Heath as I said in the beginning that I am doing my thesis on face recognition using backpropagation neural network. I would be using for now the network on the features already extracted from the face i.e. locations of the eyes etc. (I am using BioID Face Database that had 40 feature already extracted from the face).
The reason I am saying all this because I am requesting you to become my supervisor. If you said yes than please send me your contact information upon which I can formally request you to become my supervisor on the letter head of my university.

Sir i am requesting you because i want that thesis to be a quality work.
I had currently completed my course work of my MS-IT (but thesis is remaining).
From Adeel Raza

Sir you had to be MS to be my supervisor (that the restriction from the university). And by contact information I doesn’t mean yours home address but the university in which you are teaching.

On Mar 25, 5:05 am, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir you are great.

Aw shucks.

> I finally made it. I had pasted the final version of the
> element wise Backpropagation neural network code.
> Of course now I will convert it into matrix form. (Because
> until now I had been using almost any Matlab feature. This
> could well be converted into C++ (or written in ).
>
> According to the textbooks, Alpha range is from 0 to 1.

Don't believe textbooks. Based on that statement I would
automatically try the the interval 0 to 2.

> So using alpha range from 0.02 to 4, doesn’t exists
> for me. Yet alone could be reached by me. That seriously
> thinking out side the box.

You have to do that. Only by doing that did I discover those
pesky factors of 1/2. Upon further consideration of the
equations in my tutorial, I've decided that the 1/2 in din
should be changed to 2 and the 1/2 in dj should be changed
to 1 (the factor din already has the 2).

I then reran alpha = 0:0.1:4, (100 trials each) and plotted

1. Maxerror scatter plot (error goal = 0.05)
2. Maxerror statistics
3. Pct convergence
4. Final epoch scatter (epochmax = 1000)
5. Final epoch statistics

Plot 2:

Define I: 0.3 <= alpha <= 1.2

Median <= 0.05 for alpha in I
Median unstable for alpha >= I+1
For alpha in I:
Mean has a min of 0.170 @ alpha = 0.3
Mean has a max of 0.287 @ alpha = 1

Plot 3:

Pctconv >= 50% for alpha in I
Pctconv has broad peak of 73% @ alpha = 0.4,0.5

Plot 5:

Median unstable for alpha >= I+1
Median has min of 231 @ alpha = 1.2
For alpha in I
Mean has min of 518 at alpha = 1.1
518 <= Mean <= 793 for alpha in I
518 <= Mean <= 593 for 0.4 <= alpha <= 1.2

> You could also block comment by select a group of
> sentences through mouse of keyboard (shift + up )
> and then press ctrl + r (to uncomment press ctrl + t ).

Nah.

Just usr the mouse right click to comment, uncomment,
and indent

> Here is the final code
> function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)
-----SNIP

> %%% Initialization of the weights

% Google Groups: greg-heath random clock

state0 = 0
state0 = 1e9*sum(clock) % Use for duplicate reruns
randn('state',state0


> %% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
> while er==0

iteration max = 1000

while er==0 & iteration < iterationmax

> %% Backpropagation of error (Training Started)
> %%% ==Step_6==================================================
> for k=1:m
> d(Tp,k) = 0;
> %%% Error Info = (t - y) * (Derivative of Second layer activation function)
> d(Tp,k) = (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k)));

D(e^2) = 2*e*De = -2*e*Dy ==> Need a factor of 2

==Step_7==================================================
> for j=1:p
> din(Tp,j) = 0;
> for k=1:m
> din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
> end
> dj(Tp,j) = 0;
> %%% Error Info = din * (Derivative of First layer activation function)
> dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));

Note: the factor of 2 in din cancels the 1/2 fron the derivative


==Step_9==Test_stoping_condition==========================
> error = sqrt((t-y).^2);

error = abs(t-y);

> iteration = iteration +1;
> if (iteration > 3000)
> break
> end

include iterationmax in while statement

Hope this helps.

Greg

On Mar 25, 5:08=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> When I started coding for the first time I made a blunder as you will see=
 in the code that I had pasted below.

What is the blunder?. I'm tired of going thru the code.

Greg

On Mar 25, 5:09=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir Greg Heath as I said in the beginning that I am doing my thesis on fa=
ce recognition using backpropagation neural network. I would be using for n=
ow the network on the features already extracted from the face i.e. locatio=
ns of the eyes etc. (I am using BioID Face Database that had 40 feature alr=
eady extracted from the face).
> The reason I am saying all this because I am requesting you to become my =
supervisor. If you said yes than please send me your contact information up=
on which I can formally request you to become my supervisor on the letter h=
ead of my university.
>
> Sir i am requesting you because i want that thesis to be a quality work.
> I had currently completed my course work of my MS-IT (but thesis is remai=
ning).
> From Adeel Raza

I'm flattered by the offer. However, Ive been retired for 6 years.
FYI :I have a PHD from Stanford, taught at Stanford in 1970 and
at Brown from 1970-75. Although I was at MIT Lincoln Laboratory
from 1976 to 2003, I declined offers to teach because teaching
and traveling to campus from the Lab would have taken 4 half
days per week from my pioneering research in NNs.

Thanks again,

Greg

PS: What University?

The blunder is I had used as many biases as there are total records (or inputs).
Sir you had been teaching when I had not being born yet. (My date of birth is 29 / 04 / 1977.)
Sir you did not to travel or any thing as being by supervisor. None of yours precious time would be wasted. Because we could contact through the net. If you are retired this doesn’t matter. Because you being my supervisor had nothing to do with you being on Job or not.
As my supervisor my duty would be to periodically send my progress work to you and concern you when I am in trouble. This period can be any time period as you wishes. (Plus you would be paid for yours services, I am sure you would know that).
I will be talking to my teacher to find the details. Sir please say yes.

On Mar 25, 8:11=A0pm, "Adeel " <neoresearc...@gmail.com> wrote:
> The blunder is I had used as many biases as there are total records (or i=
nputs).

I don't believe you did.

> Sir you had been teaching when I had not being born yet. (My date of birt=
h is 29 / 04 / 1977.)
> Sir you did not to travel or any thing as being by supervisor. None of yo=
urs precious time would be wasted. Because we could contact through the net=
. If you are retired this doesn’t matter. Because you being my superv=
isor had nothing to do with you being on Job or not.
> As my supervisor my duty would be to periodically send my progress work t=
o you and concern you when I am in trouble. This period can be any time per=
iod as you wishes. (Plus you would be paid for yours services, I am sure yo=
u would know that).
> I will be talking to my teacher to find the details. Sir please say yes.

No.

Sorry,

Greg

Sir thank you very much for your kind help. ( I am not employed, so that I can’t pay back financially). But I know a little bit and pieces of knowledge about
Databases:
MySQL, SQL Server (MS), MS Access, Oracle.
Web Development:
PHP, XML, HTML.
Programming Languages:
     Java, C++, VB.NET.
Operating System:
Fedora, Windows 9x, Windows XP.

So if you wants any thing concerning them, I would be glad to help.
So tell me how can I pay back.
I did my BBA before I joined in the computer line (education). (but i got seriouly rusted now in that field). And last but not least I had been practicing and learning Meditation (the other side of knowledge) for now more than 10 years.

I am using as many bias as there are records.
TotalRecord = 1520; %No of Records
w0 = randc(TotalRecord,m);
v0 = rand(TotalRecord,p)-beta*ones(TotalRecord,p);
chv0 = zeros(TotalRecord,p);
chw0 = zeros(TotalRecord,m);

If you look at those statements, You would also believe
(as I did, as the saying goes “Every body will believe in the end”.

On Mar 26, 7:28=A0am, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir thank you very much for your kind help. ( I am not employed, so that =
I can’t pay back financially). But I know a little bit and pieces of =
knowledge about
> Databases:
> =A0 =A0 =A0 =A0 MySQL, SQL Server (MS), MS Access, Oracle.
> Web Development:
> =A0 =A0 =A0 =A0 PHP, XML, HTML.
> Programming Languages:
> =A0 =A0 =A0 =A0 Java, C++, VB.NET.
> Operating System:
> =A0 =A0 =A0 =A0 Fedora, Windows 9x, Windows XP.
>
> So if you wants any thing concerning them, I would be glad to help.
> So tell me how can I pay back.

You can pay me back by making it a habit to help others
whereever you see a need.

Good Luck,

Greg

P.S.

% BACKORPAGATION CODE FOR XOR
%
% Modification of code from Adeel Raza
%

clear all, close all, tic, clc, q =3D 0

% Inputs and Targets (Transposed from MATLAB NNT convention)
x =3D [0 0 1 1; 0 1 0 1]';
t =3D [0 1 1 0]';
[Ntrn n] =3D size(x);
[Ntrn m] =3D size(t);

H =3D 2 % No. of Hidden nodes
errgoal =3D 0.05 % Maximum Specified Error
maxepoch =3D 1000
Ntrials =3D 100
state0 =3D 1e9*sum(clock)
randn('state',state0)
state1 =3D randn('state') % [362436069; 521288629]

for alphaindex =3D 1:40
    alpha1 =3D 0.1 + 0.1*(alphaindex-1);
    alpha2 =3D 0.1 + 0.1*(alphaindex-1);
    Alpha1(alphaindex) =3D alpha1;
    Alpha2(alphaindex) =3D alpha2;
    for trial =3D 1:Ntrials
        disp(sprintf('alpha1 =3D %4g, alpha2 =3D %4g, Trial =3D %d',...
             alpha1,alpha2,trial));

        % Weight Initializations
        % Hidden Layer
        W10 =3D randn(1,H);
        W1 =3D randn(n,H);
        dW10 =3D zeros(1,H);
        dW1 =3D zeros(n,H);
        % Output Layer
        W20 =3D randn(1,m);
        W2 =3D randn(H,m);
        dW20 =3D zeros(1,m);
        dW2 =3D zeros(H,m);

        % Convergence Parameter Initialization
        halt =3D 0;
        epoch =3D 0;

        % Backpropagation Learning Loop
        while halt =3D=3D 0 & epoch < maxepoch
            epoch =3D epoch +1;
            for Tp=3D1:Ntrn
                % Hidden
                for j=3D1:H
                    hin(Tp,j) =3D W10(j);
                    for i=3D1:n
                        hin(Tp,j) =3D hin(Tp,j) + x(Tp,i) * W1(i,j);
                    end
                    % h(Tp,j) =3D tansig(hin(Tp,j)/2);
                    h(Tp,j) =3D (2/(1+exp(-hin(Tp,j))))-1;
                end % end hidden
                % Output
                for k=3D1:m
                    yin(Tp,k) =3D W20(k);
                    for j=3D1:H
                        yin(Tp,k) =3D yin(Tp,k) + h(Tp,j) * W2(j,k);
                    end
                    % y(Tp,k) =3D logsig(yin(Tp,k));
                    y(Tp,k) =3D 1/(1+exp(-yin(Tp,k)));
                end % end output

                % q =3D q+1,figure(q) % For m =3D 1
                % subplot(2,2,1)
                % plot(epoch,y(1),'o'), hold on
                % subplot(2,2,2)
                % plot(epoch,y(2),'o'), hold on
                % subplot(2,2,3)
                % plot(epoch,y(3),'o'), hold on
                % subplot(2,2,4)
                % plot(epoch,y(4),'o'), hold on

                % Output Weight Changes
                for k=3D1:m
                    % Error coef =3D 2*(t - y) * Dlogsig
                    dk(Tp,k) =3D 2 * (t(Tp,k) - y(Tp,k)) ...
                        * y(Tp,k) * (1 - y(Tp,k)) ;
                    for j=3D1:H
                        dW2(j,k) =3D alpha2 * dk(Tp,k) * h(Tp,j) ;
                    end
                    dW20(k) =3D alpha2 * dk(Tp,k) ;
                end
                % Hidden Weight Changes
                for j=3D1:H
                    dinj(Tp,j) =3D 0 ;
                    for k=3D1:m
                        dinj(Tp,j) =3D dinj(Tp,j) + dk(Tp,k) *W2(j,k) ;
                    end
                    % Error coef =3D 1* dinj * Dtansig(x/2)
                    dj(Tp,j) =3D (1/2)*( dinj(Tp,j) *(1 - h(Tp,j)^2) );
                    for i=3D1:n
                        dW1(i,j) =3D alpha1 *dj(Tp,j) *x(Tp,i) ;
                    end
                    dW10(j) =3D alpha1 *dj(Tp,j) ;
                end
                % Output Weight Updates
                for k=3D1:m
                    for j=3D1:H
                        W2(j,k) =3D W2(j,k) + dW2(j,k) ;
                    end
                    W20(k) =3D W20(k) + dW20(k) ;
                end
                % Hidden Weight Updates
                for j=3D1:H
                    for i=3D1:n
                        W1(i,j) =3D W1(i,j) + dW1(i,j) ;
                    end
                    W10(j) =3D W10(j) + dW10(j) ;
                end
            end % End loop over training inputs

            error =3D abs(t-y) ;
            errmax =3D max(max(error)) ;
            if errmax < errgoal
                halt =3D 1 ;
            end
            errorMax(epoch) =3D errmax ;

% disp(sprintf('Epoch =3D %4g, Maxerr =3D %d',epoch,errmax));
            % disp(['Epoch =3D ', num2str(epoch), ...
            % ' Maxerr =3D ', num2str(errmax)]) ;

            % q =3D q+1,figure(q)
            % plot(epoch,errmax,'o')
            % hold on

        end % End of while loop
        maxerror(alphaindex,trial) =3D errmax;
        numepochs(alphaindex,trial) =3D epoch;
    end % End of trials loop
end % End of alpha loop
toc
tic

q =3D q+1, figure(q)
plot(maxerror,'.') % Note 4 tracks
xlabel('alphaindex'),ylabel('Maximum Error')
title(' Maximum Error Scatter Plot')

minmaxerror =3D min(maxerror')';
medmaxerror =3D median(maxerror')';
avgmaxerror =3D mean(maxerror')';
stdmaxerror =3D std(maxerror')';
maxmaxerror =3D max(maxerror')';

q =3D q+1, figure(q)
plot(Alpha1, minmaxerror,'g');hold on
legend('Minimum')
plot(Alpha1, medmaxerror,'b');hold on % Note unstable region
legend('Median')
plot(Alpha1, avgmaxerror,'k');hold on
legend('Average')
plot(Alpha1, maxmaxerror,'r');
legend('Minimum','Median','Average','Maximum',2)
plot(Alpha1,0.05*ones(1,length(Alpha1)),'r--')
xlabel('alpha'),ylabel('Maximum Error')
title(' Maximum Error Statistics')


for alphaindex =3D 1:40
    numconv(alphaindex) =3D numel(find(maxerror(alphaindex,:) < 0.05));
end

q =3D q+1, figure(q)
plot(Alpha1,numconv/100)
xlabel('alpha'),ylabel('Percent Convergence (%)')
title(' Convergence Statistics')

q =3D q+1, figure(q)
plot(numepochs,'.')
xlabel('alphaindex'),ylabel('Number of Epochs')
title(' Final Epoch Scatter Plot')

minnumepochs =3D min(numepochs')';
mednumepochs =3D median(numepochs')';
avgnumepochs =3D mean(numepochs')';
stdnumepochs =3D std(numepochs')';
maxnumepochs =3D max(numepochs')';

q =3D q+1, figure(q)
plot(Alpha1, minnumepochs,'g');hold on
legend('Minimum')
plot(Alpha1, mednumepochs,'b');hold on % Note unstable region
legend('Median')
plot(Alpha1, avgnumepochs,'k');hold on
legend('Average')
plot(Alpha1, maxnumepochs,'r');
legend('Minimum','Median','Average','Maximum')
xlabel('alpha'),ylabel('Epochs')
title(' Final Epoch Statistics')

tilefigs % Get code from MATLAB CENTRAL
toc %

% % xlabel('epoch '), ylabel('Maximum error ')
% % title('Maximum Error')
% q =3D q+1, figure(q)
% plot(errorMax), hold on
% plot(0.05*ones(1,epoch),'r')
% xlabel('epoch '), ylabel('Maximum error ')
% title('Maximum Error')
% for j=3Dq:-1:1
% figure(j)
% end
% % save('weight6040.dat','W1','W10','W2','W20')
% toc
% W1,W10,W2,W20

Hello,

Can anyone of you kindly suggest me on this
http://www.mathworks.com/matlabcentral/newsreader/view_thread/247549#637834

Many Thanks..

Sir, We are back at square one. I did not know that the algorithm work
only for “toy problems”. When I applied it on the real data. What you know no convergence. If you can use the real data i.e. x and t of any value you have with you. To prove my point I had in the following program used random value for x and t (inputs and targets).
Here is the program.
function [v,v0,w,w0] = trainUniBPEWise(v,v0,w,w0)
%Create an back propagation neural network
%+++++++++++++++++++Formula formatted text in formulas++++++++++++
% Legent of Formatted Formula text
% _ (underscore) means Subscript
% ^ (power) means Superscript
% () parenthesis is used for grouping, (openoffice uses {} instead).
%+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
%%% Nomenclature
% x Input training vector:
% x = (x1, ..... xi, ..... xn)
% t Output training vector:
% t = (t1, ..... tk, ..... tm)
% d Portion of error correction weight adjustment for w_(jk) that is
% due to an error at the output unit Y_k; also, the information about
% the error at unit Y_k that is propagated back to the hidden units
% that feed into unit Y_k
% dj Portion of error correction weight adjustment for v_(ij) that is
% due to the backpropagation of error information from the output
% layer to the hidden unit Z_j.
% alpha Learning rate
% X_i Input unit i:
% For an input unit; the input signal and output signal are the same,
% namely, x_i.
% v0_j Bais on the hidden unit j.
% Z_j Hidden init j:
% The net input to Z_j is denoted zin_j:
% zin_j = v0_j + sum(x_i*v_(ij)).
% The output signal (activation) of Zj is denoted zj:
% z_j = f(zin_j).
% w0_k Bais on output unit k.
% Y_k Output unit k.
% The net input to Yk is denoted y_ink:
% yin_k = w0_k + sum(z_j*w_(jk))
% The output signal (activation) of Yk is denoted yk:
% y_k = f(yin_k).
%%% Activation function:
% f(x) = 2/(1 + exp(-x)) -1
% with its derivative
% f'(x) = 1/2*(1 + f(x))*(1-f(x))
%
%%% Training Algorithm
% Step 0. Initialize Weights
% (set to small random values (in the range -0.5 and 0.5,
% according to Nguyen-Widrow Initialization)
% Step 1. While stopping condition is false, do Steps 2-9
% Step 2. For each training pair, do Steps 3-8
%% Feedforward:
% Step 3. Each input unit (Xi, i = 1, ...,n)
% receives input signal xi and broadcasts
% this signal to all units in the layer
% above (the hidden units).
% Step 4. Each hidden unit (Zj, j=1, ...,p) sums
% its weighted input signals,
% zin_j = v0_j + ?_(i=1)^n x_i v_(ij)
% applies its activation function to
% compute its output signal,
% z_j = f(zin_j),
% and sends this signal to all units in
% the layer above (output units).
% Step 5. Each output unit (Y_k,k=1,...,m) sums
% its weights input signals,
% yin_k = w0_k + ?_(j=1)^p z_j w_(jk)
% and applies its activation fuction to
% compute its output signal,
% y_k = f(yin_k).
%% BackPropagation of error:
% Step 6. Each output unit (Y_k,k=1,...,m)
% receives a target pattern corresponding
% to the input training pattern, computes
% its error information term,
% d_k =(t_k - y_k) f'(yin_k),
% calculates its weight correction term
% (used to update w_(jk) later),
% chw_(jk) = alpha d_k z_j,
% calculates its bais correction term
% (used to update w0_k later),
% chw0_k = alpha d_k,
% and sends d_k to units in the layer
% below.
% Step 7. Each hidden unit (Z_j,j=1,...,p)
% sums its delta inputs (from units in
% the layer above),
% din_j = ?_(k=1)^m d_k w_(jk),
% multiplies by the derivative of its
% activation function to calculate its
% error information term,
% d_j =din_j f' (zin_j),
% calculates its weight correction term
% (used to update v_(ij) later),
% chv_(ij) = alpha d_j x_i,
% calculates its bais correction term
% (used to update v0_j later),
% chv0_j = alpha d_j.
%% Update weights and baises:
% Step 8. Each output unit (Y_k,k=1,...,m)
% updates its bais and weights
% (j=0,...,p):
% w_(jk) (new) = w_(jk) (old) + chw_(jk),
% Each hidden unit (Z_j,j=1,...,p)
% updates its bais and weights
% (i=0,...,n):
% v_(ij) (new) = v_(ij) (old) + chv_(ij),
% Step 9. Test stopping condition.
 
%% ==Step_0==Initialization=================================
clc,clear
%%% Initialization of the Input / Target
    disp('Loading the input vector x and target vectors')
% x = [0 0; 0 1; 1 0; 1 1];
% t = [0; 1; 1; 0];
    x = randn(67,40) * 5;
    t = randn(67,4) * 5;
    trainRecords = size(x,1);
% ----------------------------------------------------------
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    n = size(x,2); %INPUT
    p = n/2;
    m = size(t,2); % OUTPUT
% alpha = 0.00002;
    alpha = input('enter alpha: ');
    e = 0.05; % error limit
% -----------------------------------------------------------
%%% Initialization of the weights
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randn(n,p);
        w = randn(p,m);
        w0 = randn(1,m);
        v0 = randn(1,p);
%{
    v0 = [-0.3378, 0.2771, 0.2859, -0.3329];
    v = [ 0.1970, 0.3191, -0.1448, 0.3594; ...
           0.3099, 0.1904, -0.0347, -0.4861];
    w0 = -0.1401;
    w = [0.4919; -.2913; -0.3979; 0.3581];
%}
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0; forceQuit = false;
 
%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('max err : %d', errorMax(iteration))); % Epoch : %4g, iteration,
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            zin(Tp,j) = 0;
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            yin(Tp,k) = 0;
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
             y(Tp,k) = 1/(1+exp(-yin(Tp,k))); %activation function
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
        d(Tp,k) = 0;
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
        d(Tp,k) = (t(Tp,k) - y(Tp,k)) * (y(Tp,k) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            din(Tp,j) = 0;
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
            dj(Tp,j) = 0;
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
% if (~mod(iteration,200))
% % figure
% plot(1:200,errorMax(iteration-199:iteration))
% getframe;% F(iteration) = getframe; movie(F);
% end
    iteration = iteration +1;
    if (iteration > 600)
        forceQuit = true;
        break
    end
end %% End of wile loop Step 1
erLine = ones(1,size(errorMax,2))*0.05;
clf('reset'), cla reset
plot(1:size(errorMax,2),errorMax,1:size(errorMax,2),erLine,'r')
xlabel('iteration '), ylabel('error '), title('Plot of the error')
if(~forceQuit)
    save('weight6040.dat','v','v0','w','w0')
else
    msgbox('Maximum number of Epoch reached with out convergence','No convergence')
end
end %% End of Function / File

  On Mar 26, 3:49 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir, We are back at square one. I did not know that the algorithm work
> only for “toy problems”. When I applied it on the real data. =
What you know no convergence. If you can use the real data i.e. x and t of =
any value you have with you. To prove my point I had in the following progr=
am used random value for x and t (inputs and targets).

1. We?
2. Random inputs and outputs are real data?

The net tries to approximate some underlying
deterministic function y =3D f(x) given noisy examples
(xi,yi f(xi)+ni, i =3D 1:N) given a reasonably large SNR

SNR ~ mean(var(f(x)))/mean(var(n)) >> 10

You have SNR =3D 0 because there is no underlying
 deterministic I/O relation.

size(x) =3D [I N] % NNTBX convention
size(y) =3D [O N]

Nw =3D (I+1)*H+(H+1)*O
Neq =3D N*O

Therefore the best you can do is memorize the
training set using Nw >=3D O.

However, since the training set is random, the
net is useless for new data.

Check the MATLAB documentation and find some of their
real demo data. Otherwise check the comp.ai.neural-nets
FAQ for downloadable data.

Hope this helps.

Greg

P.S.

Use functions ZSCORE or PRESTD to standardize data
so that optimal eta1 and eta2 will not vary with the scale
of the original data.

On Mar 26, 4:36=A0pm, Greg Heath <he...@alumni.brown.edu> wrote:
> =A0 On Mar 26, 3:49 pm, "Adeel " <neoresearc...@gmail.com> wrote:
>
> > Sir, We are back at square one. I did not know that the algorithm work
> > only for “toy problems”. When I applied it on the real data=
. What you know no convergence. If you can use the real data i.e. x and t o=
f any value you have with you. To prove my point I had in the following pro=
gram used random value for x and t (inputs and targets).
>
> 1. We?
> 2. Random inputs and outputs are real data?
>
> The net tries to approximate some underlying
> deterministic function y =3D f(x) given noisy examples
> (xi,yi f(xi)+ni, i =3D 1:N) given a reasonably large SNR
>
> SNR ~ mean(var(f(x)))/mean(var(n)) =A0>> 10
>
> You have SNR =3D 0 because there is no underlying
> =A0deterministic I/O relation.
>
> size(x) =3D [I N] =A0 =A0 =A0% NNTBX convention
> size(y) =3D [O N]
>
> Nw =3D (I+1)*H+(H+1)*O
> Neq =3D N*O
>
> Therefore the best you can do is memorize the
> training set using Nw >=3D O.
>
> However, since the training set is random, the
> net is useless for new data.
>
> Check the MATLAB documentation and find some of their
> real demo data. Otherwise check the comp.ai.neural-nets
> FAQ for downloadable data.
>
> Hope this helps.
>
> Greg
>
> P.S.
>
> Use functions ZSCORE =A0or PRESTD to standardize data
> so that optimal eta1 and eta2 will not vary with the scale
> of the original data.

P.P.S.

Don't include code that is irrelevant to the discussion.

Greg

I agree with you totally. When one do thing in a hurry, things always got screwed up. So I screwed up. I try to respond as soon as possible and tried to solved the problem before you could responded but I was too late. (You had already responded).

I agree with you, and I am sorry to use the word “we”. I am back at square one.

I was never my intention to use random data and then train the network on it. I for myself used the data of “BioID Face Database” featured extracted data. But the question is how could I send it to you. Could I said to you that I am uploading a zip, please sir download it. (I believe it you would never accept that). Could I said it to you please sir download the data yourself from the net “BioID Face Database” featured extracted data. (I also believe that you would never accept that or mind that). So what else . Ahha use random data, So you would not had the trouble of downloading.
I agree with you now (when you explained to me). That was a stupid idea.

But for my defense, I remembered specifically saying that please sir “use any real data that you had”.

The “BioID Face Database” had 1520 total records. 60 % of it means 919 records. ( I am dividing the total records in 60 % for training and 40 % for testing). (And yes I did care how to divide all the records in 60 40 i.e. some person pictures are more some had less so I divide accordingly to the individual person and not ot the whole dataset.) According to
(Fausett 1994) the optimal number of hidden neuron had a formula. i.e.
W/p = e, where w is weights, p is the number of patterns and e is the max error we can tolerate. So my count of hidden units should be around 46.

Fausett, L. (1994). Fundamentals of neural networks Architectures, algorithms, and applications, Pearson education (Singapore) Pte. Ltd., Indian Branch, 482 F.I.E. Patparganj, Delhi 110 092, India.

I am sending the inputs and targets in the next message. (plus please sir can I start a new thread, because this thread had become very large).

I am converting the inputs and targets in the algorithm into bipolar using mapminmax function of Matlab to increase my chances. Also I am mixing the patterns before training (using permutation function), so safe guard against network could learn one pattern and then when the next pattern records starts it forgets the previous records. (also a suggestion by (Fausett 1994). I am also thinking of using a momentum variable to increase by chances (because of momentum the sudden jerks in the gradient can be tolerated). Next thing on my mind are to change the activation function (also a suggestion by (Fausett 1994) for some networks that doesn’t converge. Inserting a 3rd layer is next if the network would not budge. Sir I will try and try again. And I believe some thing would work.
Here is the code that I am working on.

function [v,v0,w,w0] = trainBPElementWise(v,v0,w,w0)
%% ==Step_0==Initialization=================================
clc,clear
%%% Opening of the Input / Target Files
%tic, t = toc
    directory = pwd;
    cd([directory, '\..\..\..'])
    paths
    disp('Loading the input vector x and target vectors')
    cd([directory,'\..\..\..\res\database\Face Database\ExtraFeatures\points_20']);
    FaceDataset = FaceDatabase;
    train = D6040(FaceDataset);
    trainRecords = sum(train(:));
% x = zeros(trainRecords,n); x = single(x);
% t1 = false(trainRecords,m);
    fileset = filesetter(FaceDataset, train);
    for var=1:trainRecords
        [x1(var,:),t1(var,:)] = getFileData(fileset(var,:)); %row wise
    end
    r = randperm(trainRecords);
    for i=1:trainRecords
        x(i,:) = x1(r(i),:);
        tA(i,:) = t1(r(i),:);
    end
    x1 = x;
    [x(1,:),ps] = mapminmax(double(x1(1,:)));
    for o =2:trainRecords
        x(o,:) = mapminmax('apply',double(x1(o,:)),ps);
    end
    x = single(x);
    [t,TS] = mapminmax(double(tA),-.9,.9);
    t = single(t);
    clear x1 t1 r tA thesis train var o
% ----------------------------------------------------------
%%% Enter the Architecture detail
    disp('Enter the Architecture detail');
    e = 0.05; % error limit
    n = size(x,2); %INPUT
    p = ceil(trainRecords*e); % HIDDEN Because W/p = e and 60% is 46, 100% is 76
    m = size(t,2); % OUTPUT
    alpha = input('Enter the Learning rate : ');
% alp = input('Enter the Momentum rate : ');
    beta = 0.7 * p^(1/n);
% -----------------------------------------------------------
if (nargin < 2)
    disp('weights v and w are getting initialised randomly');
        v = randn(n,p);
        w = randn(p,m);
        w0 = randn(1,m);
        vj = sqrt(sum(v.^2));
        for j =1:p
            v(:,j) = (beta * v(:,j)) ./ vj(j);
        end
        v0 = randn(1,p)-beta*ones(1,p);
end
%%%First hidden Layer
    zin = zeros(trainRecords,p);
    z = zeros(trainRecords,p);
    din = zeros(trainRecords,p);
    dj = zeros(trainRecords,p);
    chv = zeros(n,p);
    chv0 = zeros(1,p);
%%%Output Layer
    yin = zeros(trainRecords,m);
    y = zeros(trainRecords,m);
    d = zeros(trainRecords,m);
    chw = zeros(p,m);
    chw0 = zeros(1,m);
iteration =1;
er = 0; error = 0; forceQuit = false;
 
%% ==Step_1==While_stoping_condition_is_false==do_step_2-9===
while er==0
    errorMax(iteration) = max(max(error));
    disp(sprintf('Epoch : %4g, max err : %d',iteration, errorMax(iteration)));
% totalerr = 0;
%%% ==Step_2==For_Each_Training_pair===========do_Steps_3-8===
    for Tp=1:trainRecords
%% Feed forward:
%%% ==Step_3==X_get_Input=====Already_done====================
%%% ==Step_4==================================================
%%% First Layer
        for j=1:p
            zin(Tp,j) = 0;
            for i=1:n
                zin(Tp,j) = zin(Tp,j) + x(Tp,i) * v(i,j);
            end
            zin(Tp,j) = v0(j) + zin(Tp,j);
             z(Tp,j) = (2/(1+exp(-zin(Tp,j))))-1; %activation function
% z(Tp,j) = (exp(zin(Tp,j))-exp(-zin(Tp,j)))/(exp(zin(Tp,j))+exp(-zin(Tp,j))); %activation function
        end
%%% ==Step_5==================================================
    %%% Output Layer
        for k=1:m
            yin(Tp,k) = 0;
            for j=1:p
                yin(Tp,k) = yin(Tp,k) + z(Tp,j) * w(j,k);
            end
            yin(Tp,k) = w0(k) + yin(Tp,k);
            y(Tp,k) = (2/(1+exp(-yin(Tp,k))))-1; %activation function
% y(Tp,k) = (exp(yin(Tp,k))-exp(-yin(Tp,k)))/(exp(yin(Tp,k))+exp(-yin(Tp,k))); %activation function
    % totalerr = 0.5 * ((t(T,j) - y(T,j))^2)+totalerr;
        end
%% Backpropagation of error (Training Started)
%%% ==Step_6==================================================
        for k=1:m
        d(Tp,k) = 0;
    %%% Error Info = (t - y) * (Derivative of Second layer activation function)
       d(Tp,k) = (t(Tp,k) - y(Tp,k)) * ((1/2) *(1+y(Tp,k)) *(1 - y(Tp,k)));
            for j=1:p
                chw(j,k) = alpha * d(Tp,k) * z(Tp,j);
            end
            chw0(k) = alpha * d(Tp,k);
        end
%%% ==Step_7==================================================
        for j=1:p
            din(Tp,j) = 0;
            for k=1:m
                din(Tp,j) = din(Tp,j) + d(Tp,k) * w(j,k);
            end
            dj(Tp,j) = 0;
%%% Error Info = din * (Derivative of First layer activation function)
           dj(Tp,j) = (din(Tp,j) * ((1/2) * (1 + z(Tp,j))* (1 - z(Tp,j))));
            for i=1:n
                chv(i,j) = alpha * dj(Tp,j) * x(Tp,i);
            end
            chv0(j) = alpha * dj(Tp,j);
        end
%% ==Step_8==Update_weights_and_biases========================
        for k=1:m
            for j=1:p
                w(j,k)=w(j,k)+chw(j,k);
            end
           w0(k)=w0(k)+chw0(k);
        end
        for j=1:p
            for i=1:n
                v(i,j)=v(i,j)+chv(i,j);
            end
           v0(j)=v0(j)+chv0(j);
        end
    end
%% ==Step_9==Test_stoping_condition==========================
    error = sqrt((t-y).^2);
    %%% Update the stopping condition variable
    if max(max(error)) < e % W/P = e, W =weights,P =inputs, e = errors
        er =1;
    else
        er = 0;
    end
    iteration = iteration +1;
    if (iteration > 100)
        forceQuit = true;
        break
    end
end %% End of wile loop Step 1
erLine = ones(1,size(errorMax,2))*0.05;
clf('reset'), cla reset
plot(1:size(errorMax,2),errorMax,1:size(errorMax,2),erLine,'r')
xlabel('iteration '), ylabel('error '), title('Plot of the error')
if(~forceQuit)
    save([directory, '\weight6040.dat'],'v','v0','w','w0') %, 'x','tA');
else
    msgbox('Maximum number of Epoch reached with out convergence','No convergence')
end
end %% End of Function / File

Below is the input x and t of the “BioID Face Database”. T is created my me. So first person had 121 pictures and thus that many extracted records of featured extracted records from each records. So I had given him target value 1, in binary 00001. the second person had 58 pictures to this person I had given target value 2, in binary 00010 and so on i.e. given 3, 4 etc to toher person (in binary). (The pictures of the persons are not in sequence, therefore you would see 00001 and then 00010 but then 00001).

The code that I had inserted above did not divide the recordset in 60 40. I am for now concentrating on the whole record set.

x = [
154.05 102.829 223.237 102.829 156.589 175.824 219.429 173.285 129.295 90.7684 177.536 92.0379

199.752 94.5768 247.992 91.4031 125.487 107.272 138.816 105.367 169.284 103.463 209.908 103.463

237.202 106.002 253.07 107.906 188.326 132.661 177.536 140.913 199.752 141.548 187.692 171.381

187.057 190.423 186.422 224.065
144.515 89.4638 214.281 88.7519 151.634 152.111 210.721 149.975 121.022 73.802 168.007 69.5307

193.636 68.8188 237.773 75.2258 117.463 97.2946 128.853 93.0232 159.465 86.6162 197.907 86.6162

227.807 90.8875 241.333 95.8708 178.686 100.142 168.719 113.668 189.364 113.668 178.686 140.72

179.398 163.501 182.245 199.096
148.074 93.0232 219.976 95.8708 151.634 162.077 210.721 159.941 121.351 78.2225 170.855 77.3615

195.771 79.4972 242.045 79.4972 118.887 95.1589 134.548 93.7351 162.312 90.8875 202.178 93.0232

234.214 98.7184 245.604 99.4303 182.245 114.38 172.279 124.347 192.924 124.347 182.957 153.534

182.245 176.315 182.957 210.486
150.441 99.2171 224.959 100.142 157.329 174.179 217.84 169.908 123.867 85.722 174.715 85.4974

200.873 86.0884 245.893 81.5152 120.835 101.405 135.654 101.108 167.635 98.0747 207.95 97.264

240.621 102.278 252.011 100.854 187.229 127.906 176.55 135.737 199.331 136.449 188.652 164.213

190.788 188.417 190.076 222.588
150.21 96.5827 221.4 95.8708 157.329 172.044 215.704 167.772 123.312 82.7993 170.143 79.4972

198.467 83.9055 246.316 83.7686 121.734 102.99 135.26 99.4303 165.872 95.1589 208.586 95.8708

238.485 98.7184 254.147 97.2946 186.416 125.321 175.838 132.178 197.195 132.178 187.941 162.077

186.517 186.282 185.722 217.656
160.177 97.2946 234.214 100.142 161.6 173.467 228.519 174.179 133.393 85.1733 183.669 86.6162

210.009 88.7519 255.571 88.04 129.609 101.755 140.955 101.566 175.126 98.7184 217.022 98.5865

248.452 102.99 261.772 102.766 196.483 127.906 183.718 137.566 206.666 137.78 195.059 166.349

195.059 189.129 192.433 222.528
147.979 94.3986 224.247 95.1589 153.058 170.62 217.84 169.908 121.086 80.9292 172.894 81.269

198.561 82.64 246.536 78.3996 118.175 98.7184 132.46 96.6137 165.16 93.0232 207.874 93.0232

239.909 98.7184 253.435 100.854 185.64 125.292 173.889 134.035 197.329 133.45 185.485 164.353

185.805 185.57 183.935 219.859
145.939 95.1589 224.959 95.1589 153.77 172.756 219.976 169.908 122.248 83.0332 173.066 83.3606

197.093 85.2744 247.028 83.7686 119.599 102.99 130.277 100.142 165.16 96.5827 207.874 95.1589

239.909 100.142 253.435 102.99 185.377 126.12 172.991 133.601 197.339 134.862 185.805 163.501

184.381 189.129 184.751 221.214
148.786 111.533 224.959 110.821 156.617 185.57 217.84 184.146 122.446 96.5827 171.567 95.1589

200.755 97.2946 248.452 93.7351 118.175 113.668 132.413 112.244 165.16 110.109 210.721 110.109

242.045 112.244 252.687 112.444 185.805 141.432 173.703 151.399 199.331 150.687 185.805 181.298

185.805 201.231 184.704 235.609
159.465 112.956 238.485 108.685 172.279 194.824 234.214 197.672 130.277 102.278 182.245 94.447

217.128 93.0232 264.113 88.7519 131.701 120.075 143.091 117.228 177.974 114.38 222.823 112.244

256.283 109.397 274.08 104.414 202.178 159.941 186.517 165.637 217.84 165.637 200.755 189.841

203.602 222.588 203.794 250.529
144.515 111.533 224.959 111.533 148.074 190.553 218.552 190.553 118.887 97.2946 170.409 95.9266

199.331 96.5827 253.435 98.0065 114.615 119.363 127.429 115.804 160.177 111.533 206.45 112.244

242.757 117.228 256.994 120.787 182.245 135.025 169.431 148.551 195.771 149.263 182.245 179.875

182.245 209.774 182.154 245.267
134.548 112.956 213.569 113.668 140.955 194.824 210.009 191.265 103.937 102.278 157.329 95.8708

189.364 97.2946 239.909 100.854 103.225 119.363 118.175 118.651 150.922 112.244 197.195 112.244

229.942 117.228 244.892 119.363 172.991 137.873 160.888 149.975 185.805 150.687 174.414 184.146

175.126 210.486 173.392 245.718
130.277 110.109 217.84 109.397 138.108 196.248 212.145 194.112 98.2416 90.1757 159.465 85.9043

191.5 91.5994 248.452 92.3113 96.8178 113.668 111.056 110.821 150.922 107.261 200.755 107.261

237.773 113.668 252.723 113.668 175.838 138.585 163.024 151.399 190.076 152.111 177.262 190.553

175.838 214.757 176.55 256.759
133.116 140.554 231.201 138.218 141.679 241.753 219.524 239.418 98.0854 128.099 163.476 127.32

200.063 128.877 264.675 122.649 95.75 146.003 108.205 143.668 154.913 142.111 207.069 141.332

250.663 140.554 267.789 142.111 182.159 195.824 164.254 202.83 199.285 203.609 183.715 236.304

182.937 265.885 181.38 300.916
141.679 130.434 228.866 128.099 144.793 219.956 222.638 218.399 109.762 117.979 168.925 107.859

199.285 110.973 259.226 110.973 107.427 135.105 122.217 132.769 162.697 128.099 212.518 126.542

249.884 131.991 265.453 135.883 183.715 164.686 169.703 173.249 198.506 173.249 185.272 210.615

185.272 240.975 186.051 280.676
157.758 125.518 229.711 128.199 166.25 201.939 215.41 202.833 138.541 113.898 177.422 114.345

211.387 117.473 248.481 116.579 130.497 129.54 143.904 127.752 173.847 128.646 215.41 130.88

243.118 131.774 254.291 132.668 193.958 167.527 180.104 171.102 204.684 171.996 190.83 190.319

189.936 216.687 188.595 250.652
155.077 122.389 227.923 124.177 167.59 196.129 212.728 196.576 136.754 108.535 176.975 110.323

210.494 110.77 245.799 108.535 129.603 125.071 142.117 125.518 170.272 124.624 213.622 125.518

241.33 125.964 252.056 125.964 191.277 158.142 177.422 163.505 204.237 165.292 189.936 184.956

189.936 211.324 188.148 244.842
155.077 159.036 239.543 147.416 176.082 243.501 242.224 236.351 122.453 150.544 172.06 147.863

218.091 141.606 270.379 135.796 121.559 165.739 139.435 162.611 173.847 160.376 226.582 150.097

258.76 148.757 273.954 147.416 204.684 200.598 188.148 209.983 221.666 206.855 208.706 237.244

210.047 262.718 214.963 294.002
129.603 135.349 194.852 134.009 132.285 212.665 191.277 214.005 101.895 125.071 143.011 123.73

177.869 125.071 216.303 122.389 97.4259 138.031 110.833 138.031 142.117 138.925 180.551 138.031

212.281 135.349 224.348 135.349 160.44 173.784 149.714 180.487 171.613 180.934 159.993 206.855

160.44 225.178 159.546 262.718
129.603 131.327 194.852 129.987 133.179 209.536 190.383 209.983 100.107 123.73 141.67 120.155

173.847 121.048 217.197 118.367 97.8728 136.243 113.068 133.562 141.223 134.456 180.551 134.456

212.281 133.115 223.901 133.115 160.887 167.527 151.055 175.124 172.06 174.23 161.334 201.045

161.334 221.156 163.568 259.143
139.435 136.69 206.025 136.243 146.586 219.368 197.533 220.262 110.386 126.411 150.608 126.411

188.148 127.305 225.688 124.624 107.258 141.606 123.347 140.265 152.842 140.712 191.277 139.819

222.113 138.478 234.627 140.265 172.953 173.784 161.334 180.934 183.232 180.934 171.613 204.62

171.613 240.373 171.166 271.656
142.117 141.159 210.047 139.372 145.692 219.368 206.918 219.815 111.28 130.88 153.289 128.646

191.723 129.093 233.286 126.411 109.492 145.628 125.581 143.394 155.971 144.288 193.511 143.841

226.135 141.606 237.755 141.159 174.741 179.593 165.356 188.532 186.361 187.638 175.188 214.005

176.529 233.669 176.975 269.422
135.86 127.305 201.555 125.964 138.095 200.598 198.874 201.939 109.492 114.792 148.373 113.451

180.998 114.345 226.135 114.345 107.365 129.672 119.324 129.987 149.267 130.433 186.361 129.093

218.538 128.646 230.158 128.646 164.462 159.929 156.418 168.421 176.082 168.421 164.462 193.894

164.909 212.665 164.462 250.652
147.926 141.606 214.069 141.159 151.055 222.05 206.918 223.39 118.877 132.221 159.993 129.987

196.193 132.668 237.755 132.221 116.643 145.628 130.05 145.628 161.781 146.522 198.427 146.075

231.945 145.181 244.459 144.288 179.657 180.487 169.378 188.978 189.489 188.978 177.869 214.899

177.869 241.267 176.975 274.785
162.674 132.221 219.879 129.987 164.909 195.682 218.985 194.788 141.67 123.283 175.635 117.473

202.449 117.473 237.755 119.261 136.754 138.031 149.267 135.349 172.953 134.903 209.6 132.221

233.733 134.456 244.459 135.796 189.489 152.332 181.891 161.27 199.768 160.823 189.936 183.616

190.383 200.598 191.277 243.501
159.993 133.562 214.069 131.327 164.462 193.447 211.387 191.213 139.435 125.518 168.484 118.367

197.98 119.708 230.604 119.708 134.966 139.819 148.373 136.69 170.272 135.349 200.215 134.009

225.688 134.456 238.202 137.137 184.126 152.332 177.422 159.482 193.958 158.589 185.467 180.487

185.914 199.704 185.914 235.904
145.692 110.77 204.684 108.088 150.161 176.465 204.237 174.23 121.112 104.066 155.077 95.1278

184.573 96.0216 222.56 94.234 117.984 117.026 133.179 113.898 156.418 113.898 193.064 111.216

218.538 110.77 231.498 111.216 175.635 132.221 166.25 141.159 184.126 140.712 175.635 163.058

176.082 184.062 176.082 222.943
147.033 130.88 205.578 129.093 150.608 199.704 203.79 198.363 123.347 122.389 158.652 117.92

186.361 117.473 224.348 116.132 120.218 137.584 134.519 133.562 158.205 134.456 193.958 132.668

217.197 131.774 231.945 134.903 174.741 158.589 166.25 165.739 183.679 165.739 176.082 188.532

176.082 209.536 175.635 245.289
150.161 118.367 206.918 115.239 153.289 180.487 207.812 180.934 128.709 108.982 159.546 102.278

190.83 102.725 224.348 103.619 123.793 125.518 138.095 120.602 159.993 121.048 195.299 118.367

218.091 118.367 232.392 123.73 178.316 137.584 169.378 145.181 186.807 144.288 177.422 168.421

177.869 185.403 178.316 225.625
151.949 121.048 209.153 117.473 155.971 182.722 207.365 180.487 127.816 113.004 161.781 105.854

191.277 105.407 225.688 105.407 125.581 128.199 141.223 124.177 161.334 123.283 197.533 121.048

221.219 120.155 234.627 122.836 178.763 139.819 171.166 147.863 189.042 146.522 180.551 169.761

180.998 185.85 182.338 226.966
155.524 123.283 212.281 121.495 158.652 185.403 211.834 183.169 134.519 114.792 167.144 109.429

195.299 108.535 228.817 108.535 128.709 129.54 143.904 126.411 167.144 126.411 200.662 124.624

222.56 124.177 239.543 125.518 184.573 142.947 175.188 152.332 192.17 151.885 184.573 174.23

185.467 188.978 187.254 227.412
147.48 125.964 204.237 123.73 151.055 189.425 202.896 189.425 125.134 119.261 156.865 112.557

187.701 113.004 221.666 113.004 123.347 132.221 136.754 129.093 156.418 128.646 193.511 128.199

215.856 127.305 231.051 129.987 174.294 147.416 166.697 156.354 183.679 155.907 174.741 178.253

175.635 196.129 175.635 234.116
162.228 143.394 218.091 140.265 164.462 208.195 214.963 205.514 140.776 133.562 170.272 129.093

199.768 128.646 234.627 129.093 137.201 146.075 150.608 144.735 171.166 144.735 207.365 142.947

228.37 141.606 243.565 142.053 187.701 167.974 180.551 175.571 196.639 175.124 189.042 199.257

189.489 211.324 189.042 252.886
159.546 140.265 214.963 138.478 162.228 204.62 212.728 204.173 139.435 131.327 170.719 127.752

199.321 127.752 231.498 127.752 133.625 145.181 147.926 142.5 170.719 143.841 204.684 142.053

227.476 142.053 240.883 142.947 185.02 165.739 176.975 172.443 194.852 171.996 185.914 193.894

185.467 209.983 186.361 248.417
150.608 139.819 206.471 137.584 153.736 200.598 202.896 199.704 130.497 129.987 163.121 125.071

190.383 125.518 222.56 123.73 126.475 143.841 140.776 141.606 159.993 140.712 195.746 139.819

216.303 138.925 231.945 140.265 176.975 161.27 168.037 168.421 185.914 168.421 176.529 189.872

176.975 205.514 177.869 242.607
159.546 157.248 212.728 156.354 160.887 218.027 209.153 218.921 140.329 146.522 170.272 144.735

198.874 143.841 229.711 145.181 136.938 159.225 149.267 159.036 168.931 159.482 202.896 158.142

224.348 159.036 236.948 159.153 185.02 181.828 176.529 187.638 193.511 188.532 184.126 209.536

183.679 222.943 183.679 262.718
166.697 144.288 218.985 142.5 170.719 202.833 219.432 204.173 145.245 138.031 176.529 132.221

202.896 134.009 237.755 133.115 140.776 152.779 155.524 147.416 175.188 146.969 208.259 145.628

230.604 146.075 244.459 149.651 193.064 162.611 185.02 171.549 201.555 170.655 193.958 192.107

194.405 208.195 193.511 243.501
149.267 125.964 204.684 123.73 152.396 186.297 205.578 184.956 131.391 118.367 159.993 113.004

189.489 113.451 223.007 113.898 126.922 136.69 138.095 129.093 158.652 129.093 193.511 127.305

215.856 126.411 229.264 130.433 176.082 141.606 168.484 151.438 185.02 151.438 176.529 171.102

176.975 186.297 178.763 223.837
147.48 136.69 202.449 134.009 151.949 195.682 202.896 194.341 125.134 126.858 157.312 121.048

186.361 121.495 218.091 120.155 121.559 142.5 136.754 138.478 157.312 137.584 191.277 135.349

211.834 135.349 227.029 137.137 174.294 154.12 166.697 162.164 182.785 161.27 175.635 182.722

175.635 199.257 177.422 237.691
164.462 143.841 216.75 140.712 168.037 197.47 217.197 195.682 145.245 132.221 173.4 126.858

203.343 125.071 233.286 124.624 138.988 149.204 156.418 145.628 173.847 143.394 207.812 141.159

226.582 142.053 240.436 144.735 189.489 154.567 182.785 163.952 197.533 162.611 191.277 183.616

191.723 200.151 191.723 236.351
160.44 141.606 214.069 137.584 166.25 198.81 214.963 198.81 142.117 131.774 170.719 126.411

199.321 124.624 232.392 125.964 137.201 146.969 150.161 143.841 170.272 142.5 202.896 140.712

225.242 139.819 239.096 142.947 186.807 159.482 179.21 166.186 195.299 166.186 187.254 187.638

187.254 201.939 189.042 239.479
164.015 144.288 215.856 140.712 168.484 199.704 216.75 196.129 143.011 134.903 172.953 129.987

201.555 127.752 232.392 125.964 138.541 150.544 154.63 147.416 173.4 144.288 206.025 142.5

226.135 142.053 239.096 144.288 190.383 159.482 182.338 166.186 198.874 164.845 191.277 186.744

192.17 201.492 193.958 238.585
169.378 142.5 222.56 138.478 173.4 200.151 223.007 196.576 147.926 127.305 178.316 116.132

205.578 115.686 240.883 120.155 143.904 148.31 158.652 142.947 179.21 142.947 212.281 140.265

232.392 139.372 246.693 140.712 196.193 158.142 188.148 165.739 204.237 164.398 197.086 186.744

197.98 202.386 198.874 239.926
162.674 154.12 214.069 150.544 165.803 207.749 215.41 206.408 140.776 143.394 173.847 138.478

198.874 139.819 233.286 138.478 138.095 158.589 153.736 155.907 171.613 154.12 203.79 152.332

223.901 152.332 237.755 155.46 187.701 168.421 179.657 176.465 195.746 176.018 189.042 195.682

189.042 211.324 191.277 245.736
168.931 156.801 218.985 153.226 172.506 207.302 219.879 205.514 149.267 147.416 176.082 142.053

205.578 141.159 238.649 141.159 143.904 164.845 158.652 159.036 176.975 157.695 210.047 155.907

229.264 155.013 241.777 158.589 195.299 168.868 186.807 177.806 202.002 176.912 194.852 197.47

195.299 210.877 196.639 247.523
188.148 108.088 239.543 108.088 192.617 160.823 235.52 159.482 168.931 96.4685 197.98 94.6809

226.582 96.4685 256.972 96.4685 164.971 110.659 177.869 110.323 197.533 108.088 231.051 108.088

248.481 109.876 260.1 113.898 214.069 127.305 206.918 131.327 221.666 132.221 214.069 152.332

214.516 169.761 214.957 196.263
144.351 95.1278 208.259 94.234 152.842 156.354 199.768 155.907 123.211 79.0097 156.865 83.0613

189.489 83.0613 227.476 79.0391 118.877 95.1278 132.732 95.5747 155.077 95.5747 195.746 93.787

217.644 95.1278 232.839 95.1278 173.847 124.177 164.909 129.987 184.126 129.54 174.741 151.885

174.294 167.974 175.188 198.81
150.608 72.3355 211.834 71.8886 153.289 132.221 204.684 131.327 129.603 60.7159 161.334 59.3751

191.277 58.4813 231.945 60.269 128.709 77.6984 140.329 74.57 160.887 72.3355 199.768 71.8886

222.113 74.1231 237.755 77.2515 175.635 91.1056 167.144 100.491 185.02 100.044 176.082 122.389

176.082 144.288 177.422 173.784
144.798 78.5922 206.025 75.9107 150.608 138.925 201.109 136.69 123.793 68.7602 155.077 65.1849

186.361 63.8442 226.135 65.6319 122.9 84.402 135.86 80.8267 154.183 78.5922 194.852 76.3576

214.963 77.6984 230.604 80.8267 171.166 98.2561 163.568 106.3 180.104 105.854 172.506 129.987

172.953 149.204 175.635 179.593
151.055 80.3798 237.308 86.1896 151.949 176.018 223.901 179.146 115.749 55.353 175.188 56.6937

212.728 60.269 274.848 69.2071 110.386 78.1453 132.732 79.486 169.378 83.9551 218.091 87.0834

254.291 89.318 275.295 93.3401 188.595 127.752 174.741 137.584 202.449 139.819 185.02 168.868

183.232 198.363 179.657 242.16
156.865 93.3401 234.18 95.1278 158.205 179.146 227.923 180.04 125.134 71.8886 177.422 68.7602

213.175 71.4417 266.357 78.1453 118.431 94.6809 140.776 94.234 170.719 95.5747 218.985 97.3623

250.268 98.703 267.251 101.831 193.064 127.305 180.551 137.137 205.131 137.137 192.17 161.717

192.617 196.576 191.277 235.457
164.909 79.9329 245.799 84.402 164.462 163.505 243.118 166.186 134.966 59.8221 192.17 55.7999

225.242 61.6097 282.446 75.4638 133.625 81.2736 149.714 81.2736 185.914 83.9551 230.158 86.6365

264.123 87.9772 284.68 97.3623 205.578 117.473 191.277 125.071 218.985 126.411 203.79 149.204

202.002 186.744 198.874 231.435
155.971 51.7777 244.012 55.7999 153.736 144.288 239.096 146.075 117.537 27.6447 181.445 22.7287

219.879 25.4102 281.999 38.3705 110.833 52.6715 137.201 50.8839 171.613 53.5653 228.37 55.353

261.888 58.0344 285.127 61.6097 197.086 89.7649 182.785 98.2561 211.834 99.5968 194.852 126.411

195.299 164.845 193.511 214.899
152.842 45.521 243.565 48.6494 154.183 140.712 235.967 141.606 114.408 22.2818 178.763 14.6844

216.303 17.3658 281.552 32.5607 109.046 46.8617 134.966 45.9679 169.825 47.3086 227.029 49.0963

260.1 50.8839 283.786 51.7777 193.511 81.2736 180.104 91.5525 207.812 91.9994 192.617 121.048

192.617 161.717 189.042 208.642
149.714 76.8046 236.414 77.2515 155.077 166.186 233.286 163.058 114.408 52.2246 175.188 44.1803

213.622 45.9679 272.614 54.9061 113.068 78.5922 134.966 78.1453 169.825 76.3576 218.985 76.3576

252.95 79.0391 275.742 77.2515 191.277 110.323 179.21 118.367 206.471 118.367 192.17 144.288

193.064 185.403 194.405 228.306
147.926 99.1499 230.158 99.5968 158.652 190.319 233.733 188.532 118.431 83.0613 177.869 73.6762

213.175 76.3576 271.72 88.4242 115.749 105.854 136.307 101.384 171.613 101.831 220.326 102.725

253.844 102.725 273.508 109.876 193.958 133.562 182.338 144.288 207.365 143.394 192.617 169.761

194.405 206.855 194.405 248.417
143.457 96.4685 222.113 92.8932 147.033 182.722 224.348 180.04 109.046 78.1453 162.674 71.4417

200.215 68.7602 256.525 75.9107 105.917 103.172 129.156 98.703 158.652 98.2561 206.918 96.9154

237.308 94.6809 261.441 98.2561 179.657 134.903 168.037 142.053 194.852 139.819 183.232 166.633

183.679 201.492 184.126 244.395
159.546 76.8046 244.459 75.0169 164.909 171.996 243.118 168.421 120.218 57.1406 180.104 47.7555

216.303 48.2025 281.552 56.2468 117.984 82.1674 143.457 79.0391 174.741 78.5922 228.817 76.8046

260.994 75.9107 283.786 75.9107 198.874 115.239 186.361 122.836 214.963 121.495 200.215 148.757

202.449 193.894 204.237 236.798
160.44 60.7159 248.481 62.9504 159.546 152.779 241.777 152.779 123.347 38.8174 185.467 30.773

223.007 32.5607 285.574 46.8617 119.324 63.3973 144.798 62.0566 177.422 62.9504 231.051 64.2911

266.804 66.0788 287.362 70.9947 202.002 96.0216 187.254 104.513 214.963 104.513 199.768 133.562

198.874 173.337 196.639 219.815
156.418 65.1849 244.905 68.3133 154.63 160.376 239.543 161.717 117.984 42.3926 182.785 36.1359

220.326 37.0298 284.233 51.7777 113.515 68.3133 137.648 66.0788 174.741 67.4195 227.476 69.654

263.229 70.9947 287.362 74.57 198.427 102.725 183.679 112.557 213.175 113.004 197.086 140.712

195.746 188.085 193.958 232.328
156.865 70.9947 246.246 74.57 155.077 166.633 239.989 168.421 116.196 48.2025 180.551 41.9457

220.772 45.0741 285.127 57.5875 113.961 71.4417 138.541 71.8886 173.847 73.6762 228.817 76.3576

265.91 76.8046 286.915 77.6984 197.98 112.11 182.785 121.048 213.175 121.048 196.193 149.204

196.193 192.107 195.654 237.584
168.037 92.8932 247.14 92.8932 171.166 180.04 244.012 177.359 130.944 77.2515 188.148 69.2071

222.113 68.7602 281.105 76.8046 128.263 98.2561 151.055 95.1278 181.891 95.1278 231.945 94.234

263.676 94.234 284.68 95.1278 206.471 126.411 193.064 135.796 218.091 134.009 205.131 160.823

207.812 198.363 207.812 239.479
155.971 90.6587 235.074 91.5525 160.887 174.677 228.817 174.23 121.112 72.7824 177.422 66.0788

211.834 65.6319 268.145 72.7824 118.431 94.234 138.095 91.9994 170.272 92.4463 219.432 92.4463

252.056 93.3401 270.826 94.234 192.17 125.071 180.551 131.774 205.131 131.774 191.277 157.248

193.064 191.66 195.299 236.351
155.524 98.703 236.861 98.2561 156.418 184.062 236.414 183.169 119.771 78.5922 178.316 74.57

212.728 75.9107 272.167 79.0391 115.302 101.384 139.644 99.3093 173.4 100.491 220.585 98.9462

255.184 98.703 278.87 100.044 194.852 138.478 180.998 144.288 209.153 143.841 194.405 168.421

193.958 206.408 194.852 251.099
150.161 83.5082 232.392 83.9551 153.736 170.655 228.37 169.761 114.855 63.3973 170.719 56.2468

207.812 58.0344 266.357 65.1849 110.833 90.2118 134.072 85.7427 165.803 85.2958 215.41 85.2958

248.034 85.7427 269.485 89.318 189.042 118.814 176.082 125.964 202.002 125.071 189.489 151.438

189.042 188.978 185.914 235.457
164.909 108.535 244.012 105.407 173.847 197.023 239.989 194.341 129.603 90.2118 184.573 83.5082

220.772 82.6144 276.636 84.8489 126.475 113.451 149.267 110.323 179.657 109.429 228.817 108.088

258.76 106.747 280.211 105.854 202.002 144.288 191.277 152.779 216.75 150.544 204.237 176.912

206.471 218.921 208.706 260.037
159.993 96.0216 241.777 95.5747 161.781 185.403 240.436 182.722 122.453 79.486 182.338 68.7602

217.197 69.654 277.53 78.1453 119.324 100.491 142.117 96.9154 177.422 97.8092 224.795 97.8092

259.207 96.9154 279.764 97.3623 199.768 133.115 186.807 139.819 214.069 138.478 200.215 163.952

201.109 206.408 201.109 253.333
153.736 84.402 236.414 84.8489 154.63 171.996 231.051 171.996 118.431 63.8442 171.613 57.1406

210.494 59.3751 271.72 67.8664 114.408 87.0834 138.988 86.6365 169.825 86.6365 220.772 87.0834

251.162 87.0834 275.295 92.8932 191.723 122.389 179.21 128.199 205.131 128.199 191.723 154.12

190.83 190.766 190.83 235.904
155.077 87.5303 238.649 91.5525 155.524 171.102 228.37 172.443 118.431 62.9504 176.082 55.7999

214.516 58.0344 271.72 67.8664 114.408 87.0834 136.307 85.7427 173.847 85.2958 218.538 87.5303

253.844 91.1056 275.295 92.8932 193.064 121.495 180.104 126.411 205.578 126.858 191.723 153.226

191.277 191.213 190.83 235.904
154.63 136.69 235.967 139.372 162.228 223.837 229.264 225.178 122.9 110.323 176.975 113.004

216.303 112.11 271.273 111.216 116.196 133.562 141.67 136.69 169.825 138.031 223.454 138.031

255.184 138.925 276.189 139.819 192.617 187.638 179.657 189.872 207.812 189.872 193.511 216.24

193.511 246.63 192.17 282.382
151.949 79.486 237.308 78.1453 156.865 173.337 233.733 171.549 113.961 60.269 172.06 53.5653

210.94 52.2246 269.932 53.5653 110.833 84.402 134.519 81.2736 167.144 81.2736 219.879 80.8267

252.503 79.486 278.424 82.1674 192.17 121.048 177.869 129.093 206.025 127.752 193.064 156.801

193.511 194.788 193.064 238.138
147.48 63.3973 233.733 61.6097 151.949 157.248 228.37 154.12 111.727 42.8396 168.037 37.0298

206.471 35.689 269.932 41.0519 107.705 68.3133 130.497 65.6319 162.228 65.1849 217.644 63.3973

248.034 62.9504 274.401 64.2911 185.02 99.1499 173.4 108.535 199.768 108.088 187.701 136.69

189.042 176.465 189.489 221.603
150.161 87.9772 235.967 85.7427 155.524 181.381 229.711 180.04 113.961 67.4195 170.272 61.1628

210.494 61.1628 271.72 64.738 109.939 90.2118 135.413 89.7649 166.25 88.8711 219.879 87.5303

248.928 87.5303 276.636 88.8711 189.936 129.54 177.422 136.243 204.237 135.796 189.936 163.505

191.277 202.386 189.489 245.289
153.289 84.402 239.989 83.9551 156.865 174.677 233.733 172.89 118.877 59.3751 173.476 56.8923

210.94 56.2468 271.273 58.9282 114.855 87.5303 139.882 86.1896 168.931 84.8489 223.007 84.8489

254.737 85.7427 278.424 86.1896 191.723 124.624 180.104 130.88 207.365 129.093 193.511 157.695

194.405 194.788 193.511 239.479
159.546 103.172 241.777 100.938 163.121 195.682 235.967 193.001 122.9 83.0613 178.763 78.1453

214.069 77.2515 276.636 79.9329 118.431 105.407 143.011 104.96 173.4 104.513 224.348 103.619

256.525 101.384 285.127 103.619 195.746 149.204 182.785 153.226 210.047 152.332 197.086 179.146

198.427 214.899 198.427 256.015
152.842 165.739 232.839 162.611 161.781 245.736 232.392 241.267 119.771 150.991 170.272 138.925

212.728 134.456 260.994 141.159 114.408 176.912 137.648 168.868 168.037 166.186 217.622 163.945

249.375 164.845 263.676 166.186 200.662 199.704 188.148 209.089 208.462 208.819 200.215 229.2

201.109 256.015 200.029 289.074
150.161 146.522 234.627 143.394 156.865 227.859 236.861 225.178 116.643 133.562 168.037 119.708

213.622 117.473 262.335 123.73 112.174 161.27 132.732 151.438 168.931 148.31 217.197 146.969

250.715 146.075 266.804 149.204 199.321 180.487 186.807 190.766 208.162 190.657 197.98 212.665

197.533 239.032 198.427 289.533
145.245 126.858 229.264 121.942 156.418 211.324 228.817 206.408 114.408 113.451 161.781 97.3623

205.578 93.3401 258.313 106.3 110.833 136.243 129.603 132.221 161.334 126.858 211.834 123.283

244.905 124.624 261.888 126.858 190.383 146.522 180.104 158.142 200.215 159.036 191.723 178.253

194.852 228.306 194.363 270.287
146.139 128.646 229.264 121.942 156.418 211.324 228.817 206.408 113.961 116.579 161.781 97.3623

205.578 93.3401 258.313 106.3 110.833 136.243 129.156 134.009 162.674 128.199 211.834 123.283

244.905 124.624 261.888 126.858 190.383 146.522 180.104 158.142 199.768 157.248 191.723 178.253

194.852 228.306 194.363 270.287
148.373 126.858 233.286 120.155 159.993 209.536 234.18 203.726 117.09 116.579 164.015 94.6809

211.387 88.8711 263.676 105.854 114.855 140.712 133.625 132.221 165.356 125.964 216.75 122.389

248.928 123.283 264.569 124.177 193.511 144.288 183.232 156.801 203.79 156.801 194.852 179.146

197.533 226.072 194.363 270.287
149.267 125.964 232.392 121.048 159.546 205.067 233.286 201.045 117.09 116.579 164.909 95.1278

211.387 88.8711 263.676 105.854 114.855 140.712 131.391 132.221 165.356 125.964 216.75 122.389

249.375 124.624 264.569 124.177 193.511 144.288 183.232 156.801 203.79 156.801 195.299 177.359

197.086 222.943 197.98 263.612
156.418 142.053 239.543 134.903 169.378 222.943 240.436 217.134 123.347 133.562 171.613 112.557

219.432 106.3 268.592 117.473 125.581 153.226 140.329 146.522 172.06 141.606 222.56 137.584

256.972 138.031 272.614 141.159 201.109 164.398 190.83 177.359 211.834 176.465 203.79 197.917

206.025 234.563 208.754 279.836
144.798 128.199 227.476 121.495 155.077 206.408 231.945 200.151 111.28 123.73 159.993 98.703

206.918 91.5525 256.078 106.747 113.515 143.394 129.603 134.903 160.44 127.752 209.153 123.73

247.14 125.518 259.653 129.54 189.042 142.947 179.21 156.354 201.109 155.907 192.17 176.912

194.405 219.368 198.319 261.205
149.714 135.349 232.392 128.646 159.993 214.899 234.18 209.536 118.431 125.964 165.356 106.747

210.047 101.384 260.994 113.898 119.324 146.522 134.966 139.819 164.462 134.903 215.41 130.88

249.821 132.668 265.016 135.796 193.958 153.673 184.126 164.845 204.237 164.398 195.746 186.297

197.98 226.966 202.449 271.209
151.502 139.372 233.733 134.456 162.228 220.709 233.733 215.346 119.324 129.987 168.484 108.535

213.622 104.513 262.782 118.814 114.855 155.46 137.648 144.288 165.803 139.372 218.985 135.796

248.481 138.478 265.91 144.735 196.639 160.376 185.914 171.102 206.025 171.102 196.193 190.766

199.321 239.479 201.761 276.986
146.139 135.349 228.37 127.305 159.546 212.218 231.945 204.173 114.408 125.518 160.887 104.96

206.918 98.703 256.525 111.216 115.749 146.522 131.391 140.265 160.44 134.009 213.175 128.646

243.565 129.093 260.547 134.009 189.489 151.885 181.891 162.611 201.109 162.611 193.511 183.169

195.299 221.603 197.086 267.634
157.312 136.69 239.096 130.88 167.59 217.581 239.096 213.111 126.475 123.73 171.166 105.854

219.879 101.384 268.145 114.792 122.9 147.863 141.67 141.159 173.4 135.796 221.666 133.115

254.737 134.009 270.826 135.796 201.555 155.013 190.83 167.527 210.94 167.527 202.002 187.638

203.343 233.669 204.684 276.572
154.63 142.053 236.414 135.796 167.59 222.496 236.414 217.581 123.347 130.433 168.484 110.323

216.303 105.854 265.016 120.602 122.006 154.567 138.095 147.416 170.272 141.606 219.432 138.031

253.397 139.372 267.698 142.947 197.533 160.823 188.595 172.443 208.706 172.443 199.321 194.341

202.449 238.585 205.131 278.36
145.245 127.752 230.604 123.283 158.652 211.771 227.476 207.749 113.515 113.898 160.44 95.5747

209.153 91.1056 259.653 105.854 110.386 144.735 126.922 134.009 161.334 127.305 214.069 124.177

247.587 126.858 264.123 131.327 190.83 148.31 180.551 159.929 199.768 159.929 190.83 182.722

193.958 228.306 195.596 271.285
166.25 66.9726 222.56 66.5257 170.719 125.964 218.538 125.071 143.457 53.1184 171.613 52.6715

206.918 54.0123 240.436 52.2246 136.754 66.5257 153.289 66.9726 173.847 67.4195 212.728 66.9726

230.604 67.4195 245.799 66.5257 192.17 96.4685 184.573 99.5968 202.449 99.5968 193.511 121.048

193.958 136.243 193.721 163.806
175.188 88.8711 225.242 89.318 184.573 143.394 219.432 143.394 157.312 79.0391 185.02 77.6984

214.516 76.8046 241.777 76.3576 151.055 90.6587 166.25 89.7649 183.679 89.7649 218.538 89.318

234.18 90.2118 249.375 89.7649 201.555 110.77 193.958 115.686 208.685 116.179 201.109 133.562

201.109 154.12 201.555 183.616
175.635 90.6587 226.135 91.9994 181.891 142.5 221.219 142.053 158.652 80.8267 186.361 79.9329

214.069 79.9329 239.989 79.0391 152.396 91.5525 165.803 92.4463 184.126 92.4463 217.644 92.8932

234.627 92.4463 251.332 92.3969 201.555 114.792 193.379 117.857 208.722 118.306 200.215 135.349

200.215 150.991 200.215 178.7
183.232 88.8711 231.945 89.7649 191.277 139.372 227.029 138.925 165.803 78.1453 193.064 77.6984

221.666 77.2515 246.246 78.1453 161.536 93.3705 173.847 90.2118 192.617 90.2118 223.454 90.2118

240.436 90.6587 254.059 90.9666 207.365 107.194 201.031 112.935 214.951 113.166 207.365 129.093

207.812 147.863 208.259 178.253
196.193 104.513 244.905 103.619 206.471 159.929 239.543 158.589 176.529 96.4685 205.578 94.234

233.733 93.3401 261.441 94.234 174.294 105.854 186.807 104.96 205.131 105.854 237.308 105.407

252.503 105.407 266.804 104.96 223.454 127.305 215.856 132.221 230.988 132.502 222.56 149.651

223.007 171.102 223.626 195.91
171.166 59.8221 231.051 60.269 183.679 125.518 224.348 125.071 150.161 45.9679 182.338 45.0741

214.963 48.6494 252.503 47.7555 144.798 59.3751 160.44 61.6097 180.998 61.1628 220.772 62.0566

241.777 62.5035 255.883 61.5589 205.131 88.4242 195.299 92.8932 213.175 92.4463 204.237 113.898

203.79 139.819 204.684 172.443
162.228 55.353 222.56 54.9061 172.506 120.602 218.091 118.814 145.692 42.8396 173.847 42.3926

210.047 41.0519 244.459 42.8396 136.754 57.5875 151.055 57.1406 172.506 57.5875 214.069 56.2468

234.18 57.1406 248.628 57.2145 195.299 81.7205 186.361 87.0834 203.79 87.0834 195.299 108.982

195.299 129.987 196.639 166.633
195.299 81.2736 255.631 80.8267 207.812 146.075 250.715 145.181 174.294 70.5478 206.471 66.5257

240.883 66.9726 275.295 67.4195 172.06 83.5082 185.467 83.0613 205.578 82.6144 245.799 82.6144

265.91 83.5082 282.893 81.7205 226.582 105.854 218.985 112.11 236.414 112.557 227.029 133.115

228.37 155.907 231.051 192.107
169.378 115.686 231.945 114.345 179.21 178.7 227.923 181.828 152.842 103.172 183.679 100.491

220.772 101.831 250.268 100.938 148.373 116.132 159.546 117.473 180.104 116.579 222.113 115.686

243.118 115.686 254.737 115.239 202.896 143.841 193.958 149.204 213.622 150.544 203.79 169.761

204.237 195.682 201.109 224.731
166.697 110.77 227.923 107.194 180.998 170.655 224.795 172.89 149.714 96.9154 179.657 93.3401

217.644 94.6809 248.034 94.6809 145.245 116.579 158.205 112.11 177.869 110.77 219.432 108.535

237.755 108.982 254.291 114.792 201.555 138.478 191.277 142.053 210.94 142.5 201.555 162.164

201.109 184.062 199.321 214.005
165.356 120.155 234.627 117.473 179.21 177.806 224.795 179.593 149.714 97.3623 180.551 94.6809

221.666 95.5747 248.034 96.9154 142.117 118.367 156.418 119.708 176.529 116.132 222.113 115.686

242.224 117.473 252.95 115.686 202.002 142.947 192.617 147.863 212.728 148.757 202.002 169.314

200.215 193.001 197.533 223.39
164.015 113.004 227.029 110.77 176.975 175.571 222.113 178.253 142.564 102.725 178.316 92.8932

216.75 92.8932 244.905 97.8092 138.988 118.814 154.183 115.686 174.294 113.898 217.197 111.663

236.861 112.11 252.95 113.898 197.98 143.394 187.701 147.416 208.259 147.863 198.427 165.739

199.321 190.319 197.533 221.603
156.418 117.92 222.56 114.345 172.06 184.509 217.197 185.85 135.413 106.3 172.06 98.2561

209.153 98.703 244.459 100.491 130.05 124.177 147.926 120.602 168.484 118.367 213.175 117.026

233.286 116.132 246.246 117.92 194.852 150.991 184.126 154.567 205.131 154.567 194.852 176.465

195.746 198.81 193.958 230.094
163.568 116.132 226.582 113.451 177.422 179.146 222.113 180.487 143.457 105.407 180.998 97.8092

210.494 96.4685 248.034 97.8092 139.882 121.942 154.63 118.367 175.635 116.579 217.644 115.239

236.414 114.345 251.609 116.132 199.768 143.394 189.042 149.651 209.153 150.097 199.768 168.868

200.215 195.682 197.533 224.731
164.909 112.557 228.37 109.429 176.975 174.677 223.901 176.912 144.798 101.384 177.422 94.234

214.069 94.234 248.034 94.6809 140.776 120.155 155.524 114.792 176.529 113.004 218.091 110.77

239.096 111.216 254.291 117.473 200.215 139.372 189.936 145.628 210.047 146.075 199.768 165.739

199.321 188.532 197.98 219.815
169.378 113.004 232.839 111.216 180.998 176.912 227.476 178.253 150.161 101.384 186.361 97.3623

221.219 94.234 253.397 96.9154 143.228 118.93 159.546 116.132 180.551 114.345 223.454 113.004

244.459 113.898 256.972 117.026 205.131 144.288 194.852 147.863 214.963 148.31 205.131 167.08

204.684 191.66 201.555 221.156
164.462 117.92 229.264 115.239 177.869 180.04 224.348 181.381 141.67 109.429 179.21 97.8092

210.494 96.9154 252.503 104.066 139.882 123.73 153.289 121.942 178.316 117.026 218.538 116.132

239.096 117.473 252.056 118.367 198.427 143.394 189.042 148.31 209.6 149.204 199.768 170.208

199.768 195.682 198.427 226.966
165.356 114.792 229.711 112.557 175.635 178.253 221.666 180.487 139.435 108.982 177.422 99.1499

208.706 97.3623 248.481 101.831 139.435 120.155 151.949 118.367 175.188 116.132 214.963 114.345

238.202 114.792 251.162 115.686 195.746 144.735 186.361 148.757 207.365 150.097 196.193 169.761

195.299 191.66 193.958 223.837
165.356 113.004 230.158 111.216 179.21 178.253 223.007 180.487 142.564 102.725 181.891 96.9154

213.175 95.5747 250.715 96.9154 141.223 117.026 155.077 115.239 176.529 113.004 220.326 111.663

239.543 113.451 252.056 117.473 201.109 144.288 190.83 147.416 210.94 147.863 200.215 167.527

199.768 190.766 197.086 222.05
164.015 118.367 228.37 115.686 176.975 185.403 222.113 186.297 138.095 113.004 181.445 103.619

215.856 98.2561 250.268 100.938 137.774 124.976 152.842 121.495 175.188 118.814 219.432 117.026

239.096 117.473 253.844 118.367 198.427 148.31 189.042 152.779 208.706 154.12 198.427 175.124

198.427 197.917 196.639 232.328
167.59 111.216 231.945 110.323 182.338 177.806 224.795 180.04 144.798 105.407 187.701 96.9154

218.538 94.6809 254.737 98.2561 143.143 117.025 157.312 113.898 180.998 113.004 222.56 112.11

244.012 112.557 255.631 113.004 202.896 142.947 194.405 146.969 214.069 148.31 203.343 168.421

203.343 188.978 200.215 226.519
164.462 115.686 230.604 113.898 180.551 179.593 221.666 182.275 144.798 105.407 184.573 99.5968

219.432 96.9154 251.609 100.938 141.223 120.602 154.63 117.92 177.869 117.026 220.772 115.686

241.777 115.686 254.737 118.814 199.768 148.757 190.83 151.885 212.281 152.332 201.555 171.549

201.555 195.235 198.874 229.647
169.378 116.132 235.074 114.792 182.785 182.275 228.817 184.509 146.139 109.876 187.254 100.938

217.644 98.2561 257.866 101.384 142.564 121.048 159.993 118.367 182.338 117.026 225.688 116.132

246.246 116.132 258.76 117.92 205.578 148.757 195.299 151.438 215.856 153.226 205.131 171.996

204.237 197.47 202.002 228.306
163.121 121.048 227.923 119.708 176.529 187.638 222.56 189.872 139.688 110.635 180.104 104.513

213.622 103.172 249.821 105.854 134.684 125.325 151.055 123.73 174.741 121.048 216.303 120.602

238.202 121.048 252.95 121.495 197.533 155.46 187.254 157.695 210.494 158.589 197.98 181.381

197.98 204.173 196.639 231.882
162.674 117.026 227.029 115.239 175.635 182.722 223.454 184.062 140.776 108.088 181.445 101.384

213.622 98.2561 251.162 106.747 137.201 124.177 152.396 120.155 176.082 117.92 216.303 116.579

239.096 117.473 252.056 121.495 196.639 151.438 187.701 156.354 208.706 155.013 195.299 176.018

196.193 196.129 193.958 227.859
150.161 197.917 213.622 201.492 155.524 257.802 204.684 261.824 127.816 183.169 167.144 179.593

194.405 181.828 237.755 186.297 121.112 198.363 136.754 201.492 159.993 199.257 202.002 201.939

227.923 205.067 237.755 206.408 176.082 241.267 168.484 239.032 189.042 241.714 180.998 256.461

180.551 277.019 178.316 304.728
138.095 198.81 200.662 201.939 144.798 258.249 192.17 262.271 115.749 184.956 155.077 180.04

183.679 182.275 220.326 186.297 109.046 201.492 126.028 201.492 149.267 199.704 190.383 201.939

217.644 204.62 230.158 207.749 163.568 239.479 155.077 239.479 176.529 240.82 166.25 254.674

165.803 270.763 164.462 300.705
172.953 104.96 237.755 108.535 180.551 164.398 227.029 169.314 149.267 91.5525 190.383 88.8711

218.538 89.7649 261.888 92.4463 143.011 108.982 162.674 107.194 183.679 105.407 227.029 107.641

250.268 110.77 268.592 116.132 200.215 150.991 192.17 147.863 212.281 149.651 203.79 163.058

201.555 183.169 198.427 209.983
158.205 114.345 223.454 114.792 169.825 172.443 216.303 175.571 134.966 99.5968 174.741 97.8092

202.449 98.2561 246.693 97.3623 133.179 113.451 148.373 116.579 167.144 113.451 214.963 113.898

235.074 116.132 254.737 112.11 187.254 161.717 180.104 161.27 200.662 158.589 191.723 170.655

192.617 194.341 192.17 218.474
166.697 103.172 229.711 105.407 172.953 162.164 218.985 166.186 140.776 90.6587 183.679 86.1896

211.834 86.1896 252.056 90.6587 136.307 106.3 153.289 105.407 176.975 103.619 218.985 105.854

242.671 108.088 260.547 110.77 190.83 147.863 184.126 146.075 204.237 147.863 195.299 159.929

194.852 178.7 192.617 206.408
163.121 111.663 228.37 113.004 172.506 171.549 221.219 174.677 140.329 99.5968 179.657 96.9154

213.175 96.4685 250.715 99.1499 139.882 116.132 150.161 115.239 173.4 113.004 218.985 113.898

243.118 115.686 257.866 109.429 192.617 159.036 182.338 158.142 204.684 155.46 193.958 169.761

194.852 185.403 196.193 216.24
160.44 124.177 227.029 129.54 167.59 180.934 216.303 188.085 136.754 108.088 176.529 109.876

215.41 113.451 255.631 116.579 130.944 117.473 148.82 125.071 171.166 125.071 219.432 129.987

239.096 133.115 257.634 131.041 187.701 173.784 176.975 170.208 200.215 171.996 191.277 183.616

189.489 195.682 186.361 221.603
173.4 104.96 234.18 107.194 181.891 163.058 227.029 167.08 151.502 91.9994 188.148 84.8489

215.41 85.7427 256.525 92.8932 146.586 109.429 163.121 107.194 183.232 105.407 226.582 106.747

250.268 110.77 265.91 111.663 200.215 146.969 190.83 145.181 212.281 145.181 204.237 159.482

203.343 182.275 200.215 207.749
170.719 113.451 231.498 115.239 180.104 170.655 224.348 173.337 149.714 97.3623 189.936 96.9154

214.069 95.1278 253.844 99.1499 141.67 117.026 159.993 114.792 182.338 113.451 221.219 115.686

243.565 117.026 261.888 120.602 195.746 158.142 188.595 155.46 210.494 155.013 200.215 167.974

201.109 187.191 199.321 214.005
168.037 109.876 231.498 113.004 177.869 169.761 223.901 173.784 147.033 96.9154 184.573 91.5525

215.856 92.4463 255.184 98.703 140.776 114.345 158.652 112.557 179.657 110.323 223.007 112.557

246.246 115.686 262.335 117.92 194.405 153.673 187.254 154.12 207.365 152.779 199.768 167.974

200.215 187.638 197.533 214.005
173.847 110.77 237.308 115.239 182.338 169.761 226.135 174.23 151.055 96.0216 184.126 93.3401

218.538 93.3401 256.972 97.8092 143.457 113.898 162.674 113.004 182.785 111.216 228.817 115.686

249.375 118.367 266.357 121.048 197.98 156.801 190.383 155.013 211.387 155.013 202.896 166.633

202.896 188.532 201.555 210.877
167.59 106.3 233.733 110.77 172.953 165.292 227.476 169.761 144.798 91.9994 187.701 89.318

218.985 90.2118 257.866 95.1278 136.754 108.982 156.418 109.876 178.316 107.641 223.454 110.323

246.246 113.451 266.357 117.026 194.852 152.332 186.361 153.226 208.706 153.673 200.215 167.08

199.768 181.828 197.086 215.346
185.914 109.876 247.14 113.898 191.723 164.845 234.627 171.102 163.948 96.732 204.684 94.6809

228.817 94.234 268.145 101.831 162.489 111.445 176.529 111.216 196.193 110.323 237.308 113.451

258.76 116.579 275.742 122.389 208.259 154.12 200.215 151.885 221.219 152.779 211.834 163.505

212.728 184.062 212.728 206.408
177.422 95.1278 235.967 97.3623 183.679 149.651 226.582 153.673 156.418 81.7205 191.723 75.4638

214.963 78.5922 257.866 84.8489 151.949 98.703 167.144 96.4685 186.361 95.1278 226.135 97.8092

248.928 100.044 264.569 102.725 200.215 132.221 193.958 132.221 211.834 133.562 204.684 144.288

203.79 165.739 201.555 188.978
180.551 100.491 240.883 103.172 188.148 157.248 232.839 160.823 158.652 87.9772 193.064 83.5082

224.348 86.6365 263.229 90.6587 152.105 103.372 169.825 102.725 191.277 100.938 231.051 103.172

253.397 105.407 271.72 109.429 204.237 143.841 196.639 141.159 216.75 142.5 209.6 151.885

208.259 171.549 205.131 195.682
177.422 104.96 237.755 108.982 184.126 159.482 231.498 164.845 154.63 93.3401 191.723 88.4242

221.219 89.7649 258.313 95.5747 149.196 107.726 164.909 107.194 187.701 105.854 227.029 108.088

250.715 111.663 268.145 114.345 202.002 148.757 192.617 145.181 214.516 146.075 206.025 158.589

204.684 172.89 202.896 202.833
188.148 104.513 249.375 108.535 193.958 163.058 237.308 167.08 167.59 91.5525 206.025 88.8711

231.498 89.318 272.167 96.0216 157.484 106.171 176.082 106.3 198.874 105.407 238.649 108.982

262.782 111.663 278.424 114.345 213.175 145.628 204.237 143.841 226.135 145.181 216.303 159.482

214.516 174.23 210.494 202.386
182.785 102.278 242.671 105.854 189.489 159.036 234.18 163.058 162.674 89.7649 197.98 87.9772

232.392 90.2118 265.91 94.234 153.032 104.431 172.06 104.96 193.958 104.066 234.18 106.3

253.844 108.982 272.167 113.004 208.259 145.181 199.321 142.053 220.772 143.394 211.387 156.354

211.387 170.208 209.153 201.045
186.361 83.9551 245.799 87.5303 185.02 131.774 243.565 138.925 164.015 68.7602 201.555 62.0566

231.051 63.3973 267.251 73.6762 155.971 87.0834 172.506 85.7427 195.746 83.5082 236.414 87.0834

260.547 90.6587 274.401 93.3401 210.047 120.155 199.768 118.367 223.901 119.261 212.281 130.88

210.494 150.544 207.746 182.472
176.975 106.3 238.202 108.982 184.573 167.08 231.945 168.868 156.766 93.1192 193.064 87.5303

223.901 91.5525 260.1 95.1278 149.714 110.323 165.803 109.429 188.595 106.3 231.945 108.982

252.056 111.216 268.592 113.451 203.343 149.651 195.746 149.204 214.516 147.863 207.365 161.717

206.918 179.593 208.259 207.302
174.741 109.876 235.967 113.451 182.785 168.421 228.817 173.337 154.183 96.4685 189.489 92.8932

221.219 96.0216 259.653 98.703 147.926 114.792 165.356 112.11 186.361 111.216 226.582 113.451

248.481 116.132 268.145 120.602 198.874 154.12 192.17 150.991 214.069 151.885 204.684 165.739

204.684 180.487 202.449 209.089
179.21 104.513 241.777 108.088 185.467 162.611 234.18 169.314 159.546 90.2118 196.193 86.1896

223.901 89.7649 266.357 96.4685 152.396 111.663 168.931 107.194 191.723 105.407 231.498 107.641

254.291 112.557 270.826 116.579 203.79 148.31 196.193 146.969 217.197 146.075 208.706 160.823

207.365 176.018 206.025 207.749
180.104 104.513 241.33 108.535 188.595 158.142 232.392 163.058 159.099 91.9994 194.405 87.9772

225.688 90.6587 265.463 95.5747 151.949 110.77 169.825 107.194 191.277 104.96 231.498 108.535

253.397 110.77 271.72 118.367 205.131 146.075 195.746 143.841 217.644 145.181 209.6 158.142

207.365 171.102 204.237 200.598
185.914 113.004 249.821 117.473 193.511 171.549 238.202 178.253 164.015 98.703 202.002 98.703

228.817 100.044 268.145 103.172 156.418 116.132 174.741 114.792 197.086 113.898 239.096 117.473

261.441 120.602 279.764 126.411 210.494 158.589 202.449 157.695 223.007 158.589 214.516 171.549

214.963 190.766 210.94 217.581
184.126 114.345 247.587 117.473 189.936 168.421 236.414 174.23 162.674 101.384 200.662 99.1499

227.476 100.491 265.91 102.725 155.077 119.261 173.4 116.132 195.299 115.239 235.52 117.92

258.76 121.048 275.295 126.411 209.153 162.611 199.321 156.801 224.348 155.46 210.047 186.744

211.834 166.186 208.259 213.558
178.316 116.579 241.33 118.814 187.701 174.23 233.733 176.912 157.312 102.725 194.405 100.491

224.795 99.1499 264.123 102.725 149.267 123.283 167.144 119.261 189.489 116.579 231.945 118.367

251.609 121.048 272.167 126.411 206.025 159.482 196.639 159.036 219.432 158.589 209.6 171.996

210.047 187.638 210.047 217.581
176.975 121.048 239.989 123.283 187.254 179.593 229.711 182.722 154.183 108.088 193.511 106.3

223.007 105.854 261.888 108.088 147.033 129.093 165.356 122.389 188.148 121.048 230.158 122.836

253.397 125.518 271.72 130.88 202.002 165.739 194.405 163.505 218.091 162.611 206.918 176.465

207.365 197.917 207.365 222.943
173.4 117.92 237.308 120.602 181.891 176.912 228.37 181.828 150.608 104.96 189.489 104.066

216.303 105.854 259.207 107.194 149.267 124.177 161.334 120.155 183.679 119.261 225.242 121.495

251.162 123.283 271.273 125.518 197.98 164.845 190.83 162.611 211.387 163.058 203.343 175.124

204.684 189.872 204.311 215.591
183.679 108.535 245.799 113.004 191.723 168.868 235.967 173.784 162.228 94.234 196.639 91.1056

228.37 93.787 264.123 97.3623 155.077 113.451 172.506 111.663 193.958 109.429 236.861 112.557

258.313 116.132 275.295 119.708 209.153 151.438 200.215 150.097 223.454 150.544 212.728 163.505

211.834 187.191 209.6 211.324
180.104 113.451 243.118 117.026 191.277 171.996 235.967 174.677 159.099 99.5968 195.299 96.9154

224.348 96.4685 263.229 101.384 156.865 117.026 168.484 116.132 191.277 114.345 234.18 116.579

256.525 119.708 273.061 123.73 205.578 155.46 197.533 156.354 218.985 156.354 210.047 170.208

210.494 189.425 208.259 214.899
172.953 116.132 235.967 118.814 184.126 176.018 226.135 180.934 151.055 101.831 190.383 99.5968

216.75 100.938 256.972 103.172 148.82 121.495 162.228 118.814 183.679 117.473 228.37 119.261

247.587 121.495 268.145 125.964 198.427 159.929 191.723 159.036 210.94 158.589 203.79 172.443

203.79 194.341 203.79 217.581
182.785 6.1931 264.123 9.3215 189.489 86.6365 251.162 88.4242 162.228 -7.2141 211.834 -9.4487

240.883 -7.661 286.915 -5.8734 151.055 9.3215 167.59 5.7462 197.98 7.9808 246.246 9.3215

279.317 12.0029 292.725 17.3658 215.856 53.1184 203.343 58.9282 234.627 59.8221 217.644 86.6365

217.197 104.066 216.75 135.796
197.086 98.2561 238.202 99.1499 204.684 141.606 233.286 141.606 181.891 91.1056 207.107 88.958

225.272 90.2505 252.056 91.9994 180.104 101.384 187.701 100.044 203.79 98.2561 231.498 99.5968

244.459 99.5968 254.291 101.831 219.432 117.026 212.281 121.495 224.348 121.495 217.644 134.456

218.091 149.651 217.644 171.549
180.551 118.367 229.711 118.367 181.891 166.186 231.051 163.952 160.44 108.088 189.936 107.194

214.963 108.535 246.693 110.323 160.44 121.942 170.272 119.261 188.148 119.261 219.432 120.155

239.989 120.155 254.737 123.283 202.449 146.969 196.193 149.651 213.622 150.097 204.684 164.398

205.578 180.934 206.025 204.62
145.692 97.8092 194.852 98.2561 151.949 153.226 191.277 152.779 124.24 92.8932 156.418 86.6365

182.338 88.8711 214.516 91.5525 122.453 104.513 137.201 99.1499 153.736 99.5968 186.807 100.491

204.684 100.938 221.666 103.619 171.166 122.389 164.015 127.752 179.533 128.167 170.719 144.288

171.166 163.058 171.166 190.766
140.329 97.3623 190.383 99.5968 143.904 155.907 183.232 158.589 122.006 86.6365 151.949 87.5303

178.763 89.318 209.153 91.1056 119.771 102.725 129.603 98.2561 148.82 98.703 182.785 101.831

200.215 102.725 216.303 105.854 163.568 122.836 156.418 128.646 171.613 129.54 163.568 149.204

163.121 164.845 162.674 194.341
154.183 105.407 205.131 107.194 160.44 164.398 197.98 163.952 133.625 95.1278 164.015 94.234

193.064 94.6809 224.348 96.9154 130.497 110.77 143.457 107.194 162.228 106.747 196.193 108.982

215.41 109.876 231.051 114.792 179.21 129.093 171.613 134.903 187.254 135.796 177.869 151.885

177.422 177.806 177.869 204.173
164.909 73.6762 208.259 75.4638 171.613 121.495 200.215 122.389 148.82 65.6319 174.741 66.5257

197.086 66.5257 222.56 67.4195 147.033 78.1453 157.312 74.1231 173.4 75.4638 201.109 76.3576

215.41 76.8046 229.264 79.486 186.361 95.1278 180.551 100.044 192.925 100.138 185.02 114.345

184.573 133.562 183.679 153.673
160.887 93.787 214.069 94.6809 172.506 152.779 207.812 153.226 140.329 82.6144 168.931 77.6984

203.343 76.8046 234.18 82.1674 140.776 97.8092 151.055 95.5747 170.272 94.6809 206.025 95.5747

225.242 94.234 240.436 97.8092 188.148 118.814 180.551 123.283 195.746 123.283 187.701 141.606

187.701 165.292 187.701 194.788
159.099 92.8932 207.812 95.5747 165.356 146.969 202.002 146.969 140.776 84.402 168.037 83.0613

195.746 84.8489 227.029 87.5303 136.307 100.491 148.82 95.1278 167.59 94.234 200.662 96.9154

218.091 96.9154 233.286 104.066 182.338 117.92 175.188 121.942 190.383 121.048 181.891 138.031

182.338 157.248 181.891 185.85
155.971 99.5968 206.918 101.831 164.462 156.354 199.768 155.907 136.754 89.318 165.886 88.2973

196.561 88.6751 226.582 93.3401 133.625 108.982 146.139 100.491 164.015 101.384 197.98 103.619

219.879 105.407 232.839 112.11 181.445 122.836 173.847 128.199 188.595 129.093 179.657 145.181

179.657 165.739 178.763 197.023
157.758 105.407 208.259 108.982 162.228 163.952 200.215 164.398 136.307 96.4685 166.697 95.1278

194.405 96.4685 227.923 99.1499 134.072 113.004 146.139 107.641 164.909 106.747 200.215 109.876

217.197 110.323 233.286 115.239 180.998 134.456 173.847 138.031 189.176 138.746 180.104 155.46

180.998 175.571 180.104 201.492
199.768 91.5525 246.246 94.6809 203.79 143.841 241.33 143.394 180.551 83.0613 207.812 81.2736

236.861 82.6144 265.91 84.8489 177.422 101.831 187.701 94.6809 207.365 93.3401 238.202 95.1278

258.313 95.5747 271.273 97.8092 222.56 113.898 215.41 117.92 229.264 118.814 220.772 133.562

221.666 152.779 221.666 180.487
200.215 92.8932 248.481 94.6809 206.471 145.181 240.883 145.181 180.551 83.9551 208.706 82.6144

238.92 81.5864 265.91 83.9551 178.316 99.1499 189.936 94.234 208.706 94.6809 240.436 95.1278

256.972 95.5747 273.508 98.703 223.007 113.004 216.303 118.814 230.604 119.261 222.56 135.796

223.454 154.567 223.007 182.722
217.197 98.2561 264.123 100.491 221.666 150.097 259.653 150.991 198.874 89.318 229.264 90.2118

252.95 91.9994 278.87 91.9994 194.852 103.172 208.706 99.1499 225.242 99.5968 256.525 101.831

271.72 101.831 287.362 105.854 244.012 123.283 236.414 127.752 249.821 128.199 241.777 145.181

242.224 158.142 241.777 180.487
209.153 95.1278 255.631 97.3623 213.175 147.863 250.268 148.757 192.617 85.7427 217.197 84.8489

244.905 86.6365 271.72 87.0834 188.595 101.831 199.321 97.3623 217.197 96.4685 249.375 97.8092

265.016 99.5968 280.211 102.278 230.604 118.367 224.795 122.389 238.563 122.775 230.604 139.819

230.604 157.695 229.711 184.509
210.047 95.1278 259.207 96.9154 217.644 147.863 252.95 146.969 193.064 87.5303 220.326 85.2958

246.246 88.8711 274.401 87.0834 193.511 100.491 202.002 97.3623 219.879 96.9154 251.609 97.8092

268.145 98.703 283.34 101.384 234.627 118.367 227.923 121.942 241.777 122.836 233.286 138.031

233.286 156.801 234.627 184.062
210.494 101.384 260.1 104.513 217.644 160.823 250.715 161.27 190.383 91.9994 220.772 91.9994

247.14 92.4463 278.424 94.6809 187.254 109.429 200.215 102.278 218.538 102.725 252.503 105.407

269.039 106.3 286.021 110.323 234.627 126.858 227.476 132.221 241.777 132.668 231.945 149.204

233.286 172.443 233.286 197.47
212.728 91.9994 260.547 94.6809 218.091 144.288 252.056 145.628 194.852 82.1674 221.666 82.1674

248.928 83.5082 277.977 85.7427 191.723 100.938 202.896 94.234 221.666 94.234 252.95 95.1278

269.485 96.9154 282.893 100.044 235.074 114.792 229.264 120.155 243.118 119.708 234.627 136.243

234.627 155.46 235.074 182.275
207.812 95.1278 255.631 96.0216 214.963 149.204 250.268 148.757 188.595 85.2958 216.303 84.402

244.905 85.2958 273.061 85.2958 189.042 100.938 196.639 97.3623 216.303 96.9154 248.481 97.3623

266.357 97.8092 281.552 102.278 231.051 118.367 224.795 123.283 238.594 122.674 230.604 139.372

231.945 157.695 232.392 186.744
168.484 85.2958 214.516 86.6365 172.506 135.796 206.918 135.349 149.714 76.3576 175.188 75.0169

201.555 75.4638 231.945 77.6984 147.926 91.1056 158.652 87.0834 175.188 86.6365 206.025 87.0834

223.454 88.4242 239.543 91.5525 188.595 104.513 181.891 109.429 195.299 109.429 189.042 123.283

188.595 143.394 189.042 171.996
181.445 98.703 229.264 98.703 190.383 150.991 230.158 148.757 162.228 89.7649 192.17 88.8711

217.644 89.318 245.799 88.8711 157.946 104.012 173.4 100.044 189.489 99.1499 222.56 100.044

238.202 100.044 248.034 99.5968 213.175 119.708 203.343 126.411 218.091 125.964 210.94 142.053

210.494 158.142 211.387 182.275
173.4 100.044 223.007 103.619 175.635 150.544 219.879 149.651 154.63 89.318 180.998 88.4242

210.047 89.7649 241.777 92.8932 154.183 101.384 165.356 100.938 181.891 100.491 216.303 103.619

231.498 104.513 245.799 104.96 196.193 121.495 189.936 127.305 204.013 127.908 196.193 144.735

196.639 159.482 197.086 185.403
175.188 94.6809 224.795 96.0216 181.891 149.204 217.644 149.651 155.077 86.6365 184.126 83.5082

212.281 85.7427 243.261 87.0506 151.949 101.831 165.356 96.0216 184.573 97.3623 215.856 98.2561

234.627 99.5968 249.375 102.278 199.377 113.806 193.023 121.605 206.918 121.942 198.874 137.137

197.98 159.929 197.533 186.744
175.188 93.787 223.901 96.9154 182.338 150.544 216.75 151.438 155.077 86.6365 184.126 83.5082

212.281 85.7427 243.261 87.0506 151.949 101.831 165.356 96.0216 184.573 96.4685 215.856 98.2561

234.627 99.5968 249.375 102.278 199.377 113.806 193.064 120.602 206.918 121.048 197.98 135.796

197.98 163.952 197.086 190.766
177.869 53.5653 258.76 56.6937 183.232 137.584 244.012 141.159 148.97 33.715 195.245 41.64

232.629 43.5482 286.915 43.2865 147.926 58.9282 160.44 53.5653 190.383 54.9061 239.989 57.1406

273.061 58.9282 289.596 65.1849 206.918 101.384 195.299 103.619 223.007 105.407 208.259 134.009

212.728 159.482 209.153 191.213
174.294 67.4195 261.441 69.2071 182.338 156.354 240.436 157.695 143.011 55.353 193.958 53.1184

231.051 57.1406 287.809 53.5653 142.117 75.9107 157.758 68.7602 189.936 68.3133 241.33 69.654

273.954 70.9947 293.618 71.8886 209.153 117.026 196.193 120.155 227.029 121.942 212.281 151.438

211.834 173.784 213.175 208.195
182.338 61.1628 268.145 63.8442 192.17 153.226 250.268 154.12 150.608 48.2025 202.449 48.2025

241.233 53.3198 296.747 52.2246 149.714 68.3133 164.909 62.9504 197.533 63.8442 252.95 65.1849

284.68 66.9726 299.428 74.57 221.219 114.792 208.259 117.473 236.861 117.92 221.219 149.204

220.326 171.549 219.432 205.961
163.568 61.6097 249.821 62.0566 171.166 153.226 234.18 153.226 134.072 47.7555 180.998 48.2025

219.89 50.0732 280.211 47.7555 134.072 72.7824 147.926 64.738 180.104 63.3973 233.286 63.8442

265.016 65.1849 287.809 71.4417 197.086 107.641 186.361 114.345 214.516 114.792 201.555 146.522

202.002 172.443 204.684 206.408
162.228 56.2468 248.034 57.5875 172.953 148.31 231.945 148.757 131.838 46.4148 181.891 42.8396

219.634 44.0353 276.636 47.3086 130.944 62.5035 144.798 58.9282 177.422 58.9282 232.392 58.4813

264.569 61.1628 281.999 63.8442 199.321 105.854 188.148 108.088 214.963 108.982 201.109 140.265

199.321 168.868 199.768 204.62
165.803 54.4592 250.715 54.9061 172.06 139.819 236.414 140.712 134.966 40.1581 185.02 39.2643

220.114 39.4551 275.742 41.9457 131.838 58.4813 148.82 56.2468 180.104 54.9061 230.604 54.9061

264.123 57.1406 281.552 61.1628 200.215 99.5968 189.489 101.831 216.303 102.278 203.79 133.562

203.343 155.46 203.79 195.235
169.825 42.8396 254.737 44.1803 177.422 127.305 244.905 128.646 141.223 29.4323 189.936 27.1978

223.127 30.4416 283.34 32.1138 139.882 48.6494 152.396 44.6272 184.573 43.7334 237.755 44.1803

270.826 47.3086 290.49 53.5653 201.555 87.5303 191.277 91.9994 219.879 92.8932 205.578 123.73

205.131 153.673 207.365 191.213
140.776 32.5607 225.688 33.4545 147.48 121.048 212.281 122.836 111.727 20.4942 158.205 22.2818

197.533 21.8349 255.184 24.5163 111.28 40.605 123.793 34.3483 155.077 34.3483 207.812 35.689

242.671 37.0298 261.888 43.7334 174.294 82.6144 162.228 85.2958 191.277 86.1896 177.869 117.92

177.422 136.69 179.281 174.247
154.63 45.9679 240.436 46.4148 159.993 134.456 228.817 134.903 126.475 37.0298 172.953 32.5607

210.668 33.5405 269.932 31.22 126.475 52.2246 138.988 48.6494 169.378 47.3086 223.007 46.8617

257.419 51.3308 275.742 53.1184 184.126 85.2958 176.529 93.3401 201.109 93.787 188.595 125.964

190.83 153.673 193.958 192.107
167.144 50.8839 253.844 54.4592 174.294 145.628 236.414 145.628 136.754 38.3705 188.148 41.9457

226.275 42.1628 283.34 44.1803 136.754 58.9282 149.714 53.1184 182.785 53.5653 236.861 54.9061

270.379 59.3751 288.256 65.6319 197.086 96.0216 188.595 103.172 216.303 103.619 201.109 135.349

201.109 164.398 201.109 199.704
229.264 114.792 290.937 118.814 231.945 174.23 279.764 177.359 212.281 97.8092 242.224 104.513

282.893 108.088 309.707 104.513 199.231 115.592 218.985 116.132 240.436 117.473 280.658 120.155

300.769 121.048 310.154 124.624 263.676 147.416 252.95 153.226 270.826 152.779 258.313 170.655

257.419 189.425 252.503 217.581
158.652 124.624 224.348 125.964 167.144 183.616 220.326 184.062 143.011 106.3 172.953 113.898

212.728 114.792 244.459 107.641 135.413 128.199 149.267 126.858 169.825 125.518 214.069 126.411

235.967 126.858 250.715 130.433 192.17 154.12 184.126 161.27 203.79 160.823 192.617 177.806

193.958 199.257 193.064 228.753
174.294 129.54 241.33 131.774 181.891 195.235 229.264 197.023 155.971 112.557 187.254 120.155

229.711 123.283 260.547 115.239 147.926 128.199 164.909 131.327 185.467 132.668 229.711 133.562

252.503 134.009 263.229 133.115 210.047 167.527 200.215 170.655 217.644 171.549 207.812 188.978

205.578 211.324 202.896 237.244
176.529 128.646 242.671 131.774 182.785 194.341 230.158 196.129 159.546 111.216 187.701 119.261

231.498 123.283 261.888 117.026 147.341 129.592 166.25 129.54 187.701 130.433 230.158 133.562

252.056 134.009 264.123 140.712 210.047 166.186 200.215 169.761 218.538 169.314 208.259 188.532

206.918 208.642 205.131 235.904
165.356 128.199 231.945 131.327 173.4 194.341 218.538 195.682 150.161 110.323 179.21 117.026

222.113 118.814 251.162 114.792 140.329 132.668 155.077 129.093 177.422 130.433 219.879 132.221

242.671 133.562 255.631 137.584 200.662 161.27 190.383 167.08 208.259 167.974 199.321 186.744

198.427 209.983 195.299 238.585
168.037 127.752 235.967 132.221 173.847 194.788 222.113 197.47 151.949 108.982 183.232 116.579

223.007 119.708 253.844 114.792 143.904 129.093 158.652 128.199 178.763 130.433 223.454 133.562

245.352 134.009 259.207 136.69 202.002 160.376 191.723 167.08 209.6 167.974 200.215 185.85

198.427 209.983 195.299 238.585
170.719 128.199 238.202 130.88 176.082 191.213 227.923 193.001 155.971 109.429 185.467 119.708

226.582 121.942 257.419 114.792 141.223 134.903 159.993 129.093 182.785 130.88 224.795 133.562

248.034 134.009 258.76 136.69 206.471 162.164 195.299 167.974 214.516 168.421 203.79 186.297

202.896 203.726 199.768 236.798
151.949 178.253 224.795 178.7 159.993 247.076 211.387 246.183 133.179 157.248 165.803 170.208

215.856 171.549 243.565 157.695 117.537 176.018 138.541 179.593 162.674 180.934 210.494 181.828

233.733 179.593 245.352 175.124 195.299 225.178 180.998 225.625 202.002 226.072 188.148 245.289

185.914 263.612 184.126 292.214
160.44 145.181 228.37 147.863 166.25 210.877 216.303 212.665 140.329 125.964 176.975 134.903

216.75 137.137 248.034 131.327 134.072 147.416 151.055 145.181 173.4 146.522 214.516 148.757

236.861 149.204 250.268 149.204 196.193 179.593 185.467 185.403 204.237 185.85 193.958 204.62

193.064 222.943 189.936 253.333
153.736 142.5 221.666 144.735 159.099 206.855 210.047 209.536 133.625 124.177 168.037 132.221

208.259 133.115 239.989 127.752 122.583 145.67 142.564 142.947 165.803 143.841 207.812 145.628

230.158 146.522 243.118 145.628 187.701 176.465 177.422 182.275 197.086 182.722 185.914 201.939

184.573 220.709 182.785 249.758
154.183 145.181 222.113 149.204 159.993 210.877 210.94 211.771 135.413 125.518 167.144 136.69

211.387 138.478 241.777 130.88 122.922 146.734 143.904 145.181 166.25 147.863 208.706 150.097

233.286 150.097 246.553 152.108 188.595 182.722 178.316 188.532 198.427 187.191 186.361 207.302

186.361 222.943 182.785 251.992
155.077 139.372 222.56 141.606 160.44 205.067 212.281 206.855 136.307 121.942 168.484 129.54

210.047 132.221 242.671 124.177 123.728 140.355 143.904 139.372 167.144 141.606 210.94 143.394

234.627 143.394 249.467 145.593 190.383 175.124 180.551 179.593 199.321 180.04 188.148 197.917

188.148 218.027 185.467 247.076
207.812 117.92 267.698 118.814 214.069 174.677 256.972 174.23 189.489 99.1499 220.326 108.982

257.419 109.876 284.479 100.8 179.165 117.833 196.193 118.814 219.432 119.708 254.737 120.602

277.977 119.708 285.884 118.618 246.693 146.969 234.18 154.12 252.056 153.226 240.436 167.974

240.883 188.085 237.755 212.665
180.551 116.132 244.459 116.579 187.701 177.806 235.52 178.7 162.012 99.1699 193.064 105.854

231.051 105.407 262.335 99.1499 160.44 120.602 171.166 117.473 191.723 117.026 233.286 117.92

254.291 117.92 266.804 118.367 210.94 145.181 203.343 150.991 219.432 150.991 211.834 170.208

212.281 188.978 212.281 220.709
195.299 110.323 257.419 112.11 201.109 168.421 248.034 169.761 178.763 92.4463 210.047 98.2561

245.799 100.491 275.742 97.8092 173.209 110.758 185.02 110.77 206.471 112.11 246.246 113.004

268.592 114.345 280.565 113.811 226.135 136.69 218.091 143.394 234.18 143.394 226.135 160.376

226.135 180.487 224.058 212.031
182.785 107.194 242.224 108.088 188.595 164.845 234.627 166.633 165.577 92.197 194.405 97.3623

230.158 96.4685 260.994 92.8932 158.205 110.77 172.506 108.535 193.958 109.429 231.945 109.429

250.715 109.876 267.149 111.684 210.494 131.774 202.896 138.478 218.091 138.478 210.494 156.801

211.387 172.89 210.494 204.62
180.551 112.557 242.224 112.557 187.701 171.549 234.18 171.102 164.964 95.8286 192.617 101.384

227.923 101.384 260.1 97.3623 158.043 114.21 171.166 113.898 190.383 113.898 231.051 113.898

252.503 114.792 264.569 116.132 209.6 138.925 202.449 145.181 218.091 145.181 210.047 162.164

210.494 184.956 210.494 213.558
185.914 113.898 246.246 114.792 192.617 171.996 238.202 171.996 170.272 94.234 196.639 102.278

236.414 103.619 263.229 96.0216 161.241 114.356 176.082 114.345 197.086 114.345 235.52 115.239

254.737 116.132 264.14 115.063 219.879 141.606 209.6 146.969 226.582 146.522 217.197 162.611

216.75 182.722 216.75 211.771
183.232 110.323 243.565 111.216 189.489 169.314 234.627 172.443 168.931 93.787 196.639 99.1499

232.839 99.5968 263.229 96.0216 161.241 114.356 173.847 110.323 193.958 111.663 234.18 113.004

256.525 113.451 266.804 115.239 212.728 134.456 204.684 142.5 221.219 142.053 212.728 160.376

212.281 179.593 212.728 211.771
170.272 107.194 230.158 108.535 176.529 163.505 221.219 164.845 152.396 90.6587 180.998 95.5747

218.091 98.2561 248.481 96.0216 144.797 107.627 161.334 107.194 180.998 107.641 218.985 109.876

239.096 110.323 251.332 110.691 201.109 132.221 193.064 139.372 207.812 139.819 201.109 155.907

200.662 173.337 197.98 204.62
179.657 112.11 240.436 113.898 186.361 172.89 231.051 173.784 164.909 92.4463 193.958 101.384

229.711 104.066 257.866 100.044 155.524 115.239 169.378 112.557 190.83 113.898 229.264 115.239

249.821 116.132 259.653 117.92 213.622 138.925 203.343 146.075 220.326 146.522 210.94 163.058

209.6 183.616 206.918 213.111
182.338 109.876 244.459 113.004 184.126 167.974 227.476 168.868 161.781 90.6587 188.595 97.3623

230.604 99.1499 257.419 96.0216 151.949 108.982 169.825 110.323 189.042 111.216 231.051 113.004

249.821 113.898 261.425 114.012 211.456 136.184 202.002 143.394 217.644 143.394 208.259 161.717

207.365 177.359 204.684 209.983
168.484 112.557 231.051 115.686 172.506 171.996 219.432 174.23 149.714 93.787 178.763 100.938

219.432 103.172 248.481 98.703 140.924 114.213 158.205 114.345 179.657 114.345 219.432 116.132

239.989 117.473 253.397 117.026 199.768 139.372 190.383 146.969 206.918 147.863 199.321 165.739

198.427 182.275 194.852 214.452
166.697 108.982 22

Sir, how are you.

X value did not uploaded fully, because I think of the maximum limit on the text that can be insert in one post. (And t totally did not uploaded because it was after x).
So I had inserted x and t in the code file (text file). Which I am sending to you sir at your email address at the forum. i.e. heath@alumni.brown.edu

The code that I had pasted in the previous post was doing 60 40 % division of the records, through a file (actually a set of files) that I had created. But because of the (upload/download) problem, I had changing the code in the previous code. So that there is now no need for opening all those files (text files) in which the records of the picture were recorded. Instead on there is two big matrix variables in the code for x and t.

On Mar 26, 11:49 pm, "Adeel " <neoresearc...@gmail.com> wrote:
> I agree with you totally. When one do thing in a hurry, things always got=
 screwed up. So I screwed up. I try to respond as soon as possible and trie=
d to solved the problem before you could responded but I was too late. (You=
 had already responded).
>
> I agree with you, and I am sorry to use the word “we”. I am b=
ack at square one.
>
> I was never my intention to use random data and then train the network on=
 it. I for myself used the data of “BioID Face Database” featur=
ed extracted data. But the question is how could I send it to you. Could I =
said to you that I am uploading a zip, please sir download it. (I believe i=
t you would never accept that). Could I said it to you please sir download =
the data yourself from the net “BioID Face Database” featured e=
xtracted data. (I also believe that you would never accept that or mind tha=
t). So what else . Ahha use random data, So you would not had the trouble o=
f downloading.
> I agree with you now (when you explained to me). That was a stupid idea.
>
> But for my defense, I remembered specifically saying that please sir R=
20;use any real data that you had”.
>
> The “BioID Face Database” had 1520 total records. 60 % of it =
means 919 records. ( I am dividing the total records in 60 % for training a=
nd 40 % for testing). (And yes I did care how to divide all the records in =
60 40 i.e. some person pictures are more some had less so I divide accordin=
gly to the individual person and not ot the whole dataset.) According to
> (Fausett 1994) the optimal number of hidden neuron had a formula. i.e.
> W/p =3D e, where w is weights, p is the number of patterns and e is the m=
ax error we can tolerate. So my count of hidden units should be around 46.

Why haven't you searched my posts in CSSM and CANN to find out what
I
think about how to find H? Go to Google Groups and search on

greg-heath optimal hidden

> Fausett, L. (1994). Fundamentals of neural networks Architectures, algori=
thms, and applications, Pearson education (Singapore) Pte. Ltd., Indian Bra=
nch, 482 F.I.E. Patparganj, Delhi 110 092, India.

I don't use books any more.

> I am sending the inputs and targets in the next message. (plus please sir=
 can I start a new thread, because this thread had become very large).

You need to stop and reassess. I don't have time to be your advisor.
Therefore, you need to post sparingly, without including millions of
lines of code. If you have sections of code which are a problem,
just send them. For example, section of old code and section of new
code.

> I am converting the inputs and targets in the algorithm into bipolar usin=
g mapminmax function of Matlab to increase my chances.

I don't have mapminmax. If real data has natural absolute bounds
(e.g., 0 and 2*pi
then map to -1,1 otherwise standardize. Change input binary to {-1,1}
and use
tanh hidden node activations. For classification use 1 of c {0,1}
binary output
coding.

Also I am mixing the patterns before training (using permutation
function), so safe guard against network could learn one pattern and
then when the next pattern records starts it forgets the previous
records. (also a suggestion by (Fausett 1994). I am also thinking of
using a momentum variable to increase by chances (because of momentum
the sudden jerks in the gradient can be tolerated). Next thing on my
mind are to change the activation function (also a suggestion by
(Fausett 1994) for some networks that doesn’t converge.

Inserting a 3rd layer is next if the network would not budge. Sir I
will try and try again. And I believe some thing would work.

Something will work. However, you need to stop and think for
yourself.
Always use zero mean inputs and tanh hidden nodes. One hidden layer
is sufficient. Output nodes are linear unless you need to quote
posterior
probabilities then use logsig or softmax.

I think writing your own learning code is good for understanding.
However,
Now that you've done it. Stop wasting time and use MATLAB's NEWFF.
Then you can spend more time on the real-world task you are trying to
accomplish.

> Here is the code that I am working on.

Only include modifications. Otherwise you are wasting space and
bandwidth.

> function [v,v0,w,w0] =3D trainBPElementWise(v,v0,w,w0)
> %% =3D=3DStep_0=3D=3DInitialization=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> clc,clear
> %%% Opening of the Input / Target Files
> %tic, t =3D toc
> directory =3D pwd;
> cd([directory, '\..\..\..'])
> paths
> disp('Loading the input vector x and target vectors')
> cd([directory,'\..\..\..\res\database\Face Database\ExtraFeatures\poi=
nts_20']);
> FaceDataset =3D FaceDatabase;
> train =3D D6040(FaceDataset);
> trainRecords =3D sum(train(:));
> % x =3D zeros(trainRecords,n); x =3D single(x);
> % t1 =3D false(trainRecords,m);
> fileset =3D filesetter(FaceDataset, train);
> for var=3D1:trainRecords

Change: var is a MATLAB function

> [x1(var,:),t1(var,:)] =3D getFileData(fileset(var,:)); %row wise
> end
> r =3D randperm(trainRecords);
> for i=3D1:trainRecords
> x(i,:) =3D x1(r(i),:);
> tA(i,:) =3D t1(r(i),:);
> end
> x1 =3D x;
> [x(1,:),ps] =3D mapminmax(double(x1(1,:)));

Change

> for o =3D2:trainRecords
> x(o,:) =3D mapminmax('apply',double(x1(o,:)),ps);
> end
> x =3D single(x);
> [t,TS] =3D mapminmax(double(tA),-.9,.9);
> t =3D single(t);
> clear x1 t1 r tA thesis train var o
> % ----------------------------------------------------------
> %%% Enter the Architecture detail
> disp('Enter the Architecture detail');
> e =3D 0.05; % error limit
> n =3D size(x,2); %INPUT
> p =3D ceil(trainRecords*e); % HIDDEN Because W/p =3D e and 60% is =
46, 100% is 76

greg-heath Nw Neq
greg-heath optimal hidden

> m =3D size(t,2); % OUTPUT
> alpha =3D input('Enter the Learning rate : ');
> % alp =3D input('Enter the Momentum rate : ');
> beta =3D 0.7 * p^(1/n);

Waste of time use newff

Hope this helps.

On Mar 27, 5:34 am, "Adeel " <neoresearc...@gmail.com> wrote:
> Sir, how are you.
>
> X value did not uploaded fully, because I think of the maximum limit on t=
he text that can be insert in one post. (And t totally did not uploaded bec=
ause it was after x).
> So I had inserted x and t in the code file (text file). Which I am sendin=
g to you sir at your email address at the forum. i.e. he...@alumni.brown.ed=
u
>
> The code that I had pasted in the previous post was doing 60 40 % divisio=
n of the records, through a file (actually a set of files) that I had creat=
ed. But because of the (upload/download) problem, I had changing the code i=
n the previous code. So that there is now no need for opening all those fil=
es (text files) in which the records of the picture were recorded. Instead =
on there is two big matrix variables in the code for x and t.

It is beter to use data files.

help save
doc save
help load
doc load

ascii text format is easier for me

Hope this helps.

12345678901234567890123456789012345678901234567890123456789012345
On Mar 26, 11:49 pm, "Adeel " <neoresearc...@gmail.com> wrote:

> Could I said it to you please sir download the data
> yourself from the net “BioID Face > Database”
> featured extracted data.

Yes. What is the URL? I assume it has other
info on the data?

> The “BioID Face Database” had 1520 total records.
>60 % of it means 919 records. ( I am dividing the total records
>in 60 % for training and 40 % for testing). (And yes I did care
>how to divide all the records in 60 40 i.e. some person pictures
>are more some had less so I divide accordingly to the individual
>person and not ot the whole dataset.)

How many for each person(class)?

>According to (Fausett 1994) the optimal number of hidden neuron
>had a formula. i.e. W/p = e, where w is weights, p is the number
>of patterns and e is the max error we can tolerate. So my count
>of hidden units should be around 46.

That doesn't make sense. Look again. On what principle was the
equation derived?

> Fausett, L. (1994). Fundamentals of neural networks
>Architectures, algorithms, and applications, Pearson education
>(Singapore) Pte. Ltd., Indian Branch, 482 F.I.E. Patparganj,
>Delhi 110 092, India.

I gave all my books to M.I.T.

> I am sending the inputs and targets in the next message.
>(plus please sir can I start a new thread, because this thread
>had become very large).

Certainly in length of posts!

> I am converting the inputs and targets in the algorithm into
>bipolar using mapminmax function of Matlab to increase my chances.

prestd better for realvalued inputs

>Also I am mixing the patterns before training (using permutation
>function), so safe guard against network could learn one pattern
>and then when the next pattern records starts it forgets the
>previous records. (also a suggestion by (Fausett 1994).

For classifiers probably better to present inputs in groups
of c (for c classes) Within each group the order of the c
class labels is random. This prevents large classes from
dominating the learning at the expense of smaller classes.
If the largest class has N1 patterns, then the total number
of patterns per epoch will be N1*c; i.e., the training set
will contain duplicate patterns from smaller classes.

> I am also thinking of using a momentum variable to
>increase by chances (because of momentum the sudden
>jerks in the gradient can be tolerated). Next thing
>on my mind are to change the activation function (also
>a suggestion by (Fausett 1994) for some networks that
>doesn’t converge.

This is a waste of time. The NN Toolbox has 12 learning
algorithms for this type of net. The ranks of gradient
descent and gradient descent with momentum are 12 and
11, respectively. See the NNTBX documentation online
or on the MATHWORKS website.

I recommend learning to use one or more of the best
learning algorithms to do your job. Then you can go
back and see how well the bad ones compare.

> Inserting a 3rd layer is next if the network would not
> budge.

Not recommended for newbies. I know of no reliable
way to choose the number of nodes per hidden layer.
Leave it to the experts.

>Sir I will try and try again. And I believe some
>thing would work.

Again, a waste of time. Spend the time on learning how
to use the good algorithms.

If you do use the NNTBX learning algorithms you will
have to transpose the input and output data matrices

> Here is the code that I am working on.
>
> function [v,v0,w,w0] = trainBPElementWise(v,v0,w,w0)
> %% ==Step_0==Initialization=================================
> clc,clear
> %%% Opening of the Input / Target Files
> %tic, t = toc
> directory = pwd;
> cd([directory, '\..\..\..'])
> paths
> disp('Loading the input vector x and target vectors')
> cd([directory,'\..\..\..\res\database\Face Database\ExtraFeatures\points_20']);
> FaceDataset = FaceDatabase;
> train = D6040(FaceDataset);
> trainRecords = sum(train(:));
> % x = zeros(trainRecords,n); x = single(x);
> % t1 = false(trainRecords,m);
> fileset = filesetter(FaceDataset, train);
> for var=1:trainRecords
> [x1(var,:),t1(var,:)] = getFileData(fileset(var,:)); %row wise
> end
> r = randperm(trainRecords);

Calculate the number of patterns in each class.

You need some data visualation plots and some attempt at
input variable reduction. i doubt if you need all 40 features.

Again, present patterns in groups of c containing 1 randomly
ordered pattern from each class;e.g.. 1 2 3 4 5 3 4 2 1 5 etc,


> for i=1:trainRecords
> x(i,:) = x1(r(i),:);
> tA(i,:) = t1(r(i),:);
> end
> x1 = x;
> [x(1,:),ps] = mapminmax(double(x1(1,:)));

Use prestd

> for o =2:trainRecords
> x(o,:) = mapminmax('apply',double(x1(o,:)),ps);
> end
> x = single(x);
> [t,TS] = mapminmax(double(tA),-.9,.9);
> t = single(t);
> clear x1 t1 r tA thesis train var o
> % ----------------------------------------------------------
> %%% Enter the Architecture detail
> disp('Enter the Architecture detail');
> e = 0.05; % error limit

This is a classifier. The only summary that makes sense
contains the percent error rate for each class and the
overall percent error error rate.

The learning algorithms typically minimze mean-squared-error.
The maximum error of the c outputs is a weak descriptor for
classification and should be replaced.

I recommend minimize MSE and include the percent error rate
in the result tabulation.


> n = size(x,2); %INPUT
> p = ceil(trainRecords*e); % HIDDEN Because W/p = e and
> 60% is 46, 100% is 76

greg-heath Nw Neq
greg-heath optimal hidden

Use newff and find which of the three highest recommended
algorithms is best for your data.

1. trainlm
2. trainrp
3. trainscg

Hope this helps.

Greg

On Mar 27, 9:15 pm, Greg Heath <he...@alumni.brown.edu> wrote:
> 12345678901234567890123456789012345678901234567890123456789012345
> On Mar 26, 11:49 pm, "Adeel " <neoresearc...@gmail.com> wrote:
>
> > Could I said it to you please sir download thedata
> > yourself from the net “BioIDFace> Database”
> > featured extracteddata.

Preferable to email the data in ascii format and reference
the URL for contextual info.

> > The “BioIDFaceDatabase” had 1520 total records.
> >60 % of it means 919 records. ( I am dividing the total records
> >in 60 % for training and 40 % for testing). (And yes I did care
> >how to divide all the records in 60 40 i.e. some person pictures
> >are more some had less so I divide accordingly to the individual
> >person and not ot the whole dataset.)
>
> How many for each person(class)?

Oh boy!

I was finally able to get these training set
MSE results using all of the data and a
I-H-O = 40-41-5 MLP with

Nw = (40+1)*41+(41+1)*5 = 1891
Neq = 1520*5 = 7600
r = Neq/Nw = 4.02

Nepochs MSEtrn
1000 0.072
3000 0.00676
4300 0.00229
8300 0.000796

As a reference,

MSE00 (Assumes the output is equal to the mean
of the training set) = 0.229

MSE0 (Backslash Linear classifier) = 0.1073

However, this is supposed to be a classifier.
Therefore, the only practical performance
descriptor is percent classification error
or correct classification rates... Both 5
class error rates as well as an overall
mixture error rates.

In order to do this I took a look at the 5-dim
output matrix,t, expecting to see 1-of-c coding
for 5 classes (persons).

Instead, I find 23 different target IDs coded
via 5-bit binary.

Perpexed, I found the BioID Face Recognition
database and found that there really are 23
faces with their IDs coded in 5 bit binary!

I am sure that it will be better to convert
the 5-bit binary target matrix to one of 23
23-bit binary. Otherwise you will have to do
the inverse conversion implicitly within the
code.

Have mercy on your hidden units and change
the output coding.

Keeping H = 41

Nw = (40+1)*41+(41+1)*23 = 2647
Neq = 1520*23 = 34960
r = Neq/Nw = 13.2

P.S. What is the goal of your master's thesis?

Sir How are you,

Here is the url of BioId face database.
http://www.bioid.com/downloads/facedb/index.php

Yes you are exactly right there are 23 persons pictures in the database. Minimum pictures of a person is two pictures while max pictures of a person is 150. Total is 1520.

Ok, Sir only big questions.

1. Sir does target value 1,2,3 etc are better or 00001, 00010, 00011 etc (in binary 1,2,3 ) is better. You are suggesting to convert 5 bit binary in to 23 bit . so you means I should insert 18 zeros in front of every target. (I selected 5 bit because it is the least, or you suggesting to have only one bit on for that records and all other zero).

2. If I change to 23 bit targets, what should be the number of targets. (same 41 ?).

3. If the input range is in between
Min -9.448660 and Max 354.845, should I map it -1 and 1 range.

> What is the goal of your master's thesis?
My teacher suggested that I should create my own code for the backpropagation because the Matlab function uses to much matrics in its algorithm and is cumbersome for the most part and slow. So I wasted months on the code now.
and I believe went finally I will create the code he will says that work is already done by others what new you had done ( and I will lose half of my blood with only his that sentence).
My goal is to create an efficient face recognition algorithm.