what is wrong for my polynomial regression with gradient descent?

i just want to make a polynomial regression with gradient descent ,the funtion i make regression is sin(x),but i am surprised ,if i make regression other function ,just like ax+b ax^2+bx+c,it is succeed,but for sinx,however i change the alpha,it can't succeed,what is problem about my code?

%set the data
N=20
X=(0.1:0.01:2*pi)
Y=sin(X)
r=randi(size(X),[N,1])
X_T=X(r)
Y_T=Y(r)
Y_T=Y_T+randn(1,N)/10
Y_T=Y_T'
hold on;
scatter(X_T,Y_T,'Red'); 
plot(X,Y)
% GRADIENT DESCENT
W_T = randn(1,5);     % initialize W to all zeros
N_W=W_T
V  =[ones(N,1) X_T' (X_T.^2)'  (X_T.^3)' (X_T.^4)']
alpha = 0.00000000005;    % gradient descent step size
W_C=inv(V'*V)*V'*Y_T;

% for each iteration
while 1
for j = 1:5     % looping for each parameter W(i)
    % calculate new value of W(i)
    N_W(j) = W_T(j) -alpha *sum(((V(:,j)*W_T(j)') - Y_T).* V(:,j));
end 
 if(norm(W_T-N_W)<0.001)
     break
 end
 W_T=N_W;
end
T_X=(0.1:0.1:2*pi)
T_Y=N_W(1)+T_X.*N_W(2)+(T_X.^2).*N_W(3)+(T_X.^3).*N_W(4)+(T_X.^4).*N_W(5)
plot(T_X,T_Y, 'Color','Green');
% output actual weights from formula in Red W = [5.775 0.474]'
% finish off
hold off;

[EDITED, Jan, code formatted]

Answer 1

[Matt J]

Your stopping criterion isn't a very good one

if(norm(W_T-N_W)<0.001)
   break
end

It is easy to satisfy this criterion because alpha is so small and the algorithm is taking very small steps. A better criterion would be to watch for the gradient norm to get sufficiently small, which could take a long time, because the problem is rather ill-conditioned, cond(V)=~1e4, and because you are taking very small steps.

[Matt J]

Incidentally, using MATLAB linear algebra operations, you can compute the gradient descent iterations much more efficiently,

while 1

    grad=V.'*(V*W_T(:) - Y_T);

   if(norm(grad)<.01)
       break
   end

     W_T(:) = W_T(:) - alpha *grad;

end

Answer 2

[hu]

thank Matt very much ,just look at your suggestion,i modify my code as such,it run very well,but as you say ,if you want correct weight,it will cost much more time than just polynomial fun if you use sinx fun

%set the data
N=40;
X=(0.1:0.01:2*pi);
Y=sin(X);
r=randi(size(X),[N,1]);
X_T=X(r);
Y_T=Y(r);
Y_T=Y_T+randn(1,N)/10;
Y_T=Y_T';
hold on;
scatter(X_T,Y_T,'Red'); 
plot(X,Y)
% GRADIENT DESCENT
W_T = randn(4,1);     % initialize W to all zeros
N_W=W_T;
V  =[ones(N,1) X_T' (X_T.^2)'  (X_T.^3)'];
alpha = 0.0001;    % gradient descent step size

% for each iteration
while 1
for j = 1:4    % looping for each parameter W(i)
    % calculate new value of W(i)
    N_W(j) = W_T(j) -alpha *(1/N)*sum((V*W_T - Y_T).* V(:,j));
end 
 W_T=N_W;
 grad=V.'*(V*W_T(:) - Y_T)

   if(norm(grad)<0.5)
         break
    end
  end
  T_X=(0.1:0.1:2*pi);
  T_Y=N_W(1)+T_X.*N_W(2)+(T_X.^2).*N_W(3)+(T_X.^3).*N_W(4);
  plot(T_X,T_Y, 'Color','Green');
  % output actual weights from formula in Red W = [5.775 0.474]'
  % finish off
  hold off;

[Matt J]

but as you say ,if you want correct weight,it will cost much more time than just polynomial fun if you use sinx fun

No, I did not say that it would take longer for sin(x), in general. The theory of gradient descent says that speed of convergence is determined by cond(V), alpha, and your initial point. In your case, you probably saw earlier stopping with a pure polynomial because the initial guess

W_T=randn(1,4)

happened to be closer to the final solution than for sin(x). A better initial guess for sin(x) might be its 4th order Taylor series approximation, for example,

W_T= [ -12.1567   23.6088   -9.4248    1.0000         0];