How can I use LDA (Fisher Discrimnant Analysis) to get a transformation matrix?

I know I should choose the eigenvectors with the biggest eigenvalues.But those eigenvalues are complex, many of them are NaN. In this case how should I choose the eigenvectors?

Here is what I have done so far.

function [Z,teZ,W] = myLDA(X,Y,teX,r)

[d, n]=size(X);

ClassLabel = unique(Y);
k = length(ClassLabel);
NumberOfClasses = size(ClassLabel,2);

if(nargin==3)
   r = NumberOfClasses-1;
end

nGroup=NaN(k,1);
GroupMean=NaN(d,k);
SB=zeros(d,d);        
SW=zeros(d,d);

for i=1:k
    group = (Y==ClassLabel(i));
    nGroup(i) = sum(group);
    GroupMean(:,i)=mean(X(:,group),2);
    t = X(:,group) - GroupMean(:,i)*ones(1,nGroup(i)); 
    SW=SW+t*t';
end
m = mean(X,2); 
for i=1:k
    tmp = GroupMean(:,i)-m;
    SB = SB + nGroup(i)*(tmp*tmp');
end

[V,D] = eig(SB,SW,'chol');
V = fliplr(V);
W = V(:,1:r);
Z = W'*X;
teZ = W'*teX;

I try the simple way but the results are not good for classification(I used it for handwritten digits recognition).