What exactly does "but it does not work" mean?
Your solution 1:
A = [-10,10,0];
B = [0,0,1];
C = (dot(A,B)/norm(B)^2)*B
This looks ok. If you get C = [0,0,0], the method works. A and B are orthogonal, such that the projection is zero.
Your solution 2: wrong
CosTheta = dot(A,B)/(norm(A)*norm(B));
ThetaInDegrees = acos(CosTheta)*180/pi;
Now c is a scalar, but you wanted a vector. Converting Theta in degrees is not correct here: COS works win radians. Use COSD for degerees. Improved:
CosTheta = dot(A,B) / (norm(A)*norm(B));
C = norm(A) * CosTheta * B / norm(B);
And as expected: If you insert CosTheta in the 2nd line, you get your solution 1 again.