Thread Subject:

Operations on a matrix all resulting in NaN

Good day,
I have a number of matrices that I'm working on with an m-file. All these matrices are quite large, for example one of them has the dimensions, 23983 x 3. All have 3 columns. While calculating the standard deviation of the columns using a for loop, I started to receive a value of NaN for some of the matrices. The first 10 calculated ok but the rest just kept bring out NaN. I couldn't understand it as I had picked one of these matrices at random and used microsoft excel to calculate the standard deviation and I received an answer. So I tried other operations. I tried finding the mean, then the sum of matrix but still, "NaN". I really can't figure out what is happening so any help will be appreciated.
Thanks.

First, it's simpler and quicker to use the std function to compute the standard deviations rather than a for loop.

Second, the NaNs in the result are likely to be because the data contain NaNs. So you need to look at how the original matrices are obtained, to see why they contain NaNs. If it makes sense simply to remove NaNs prior to computing the sd, it's easy - there have been quite a number of threads here recently showing how to do it, I think, so I won't go into details, but if you can't find one say so here.

"David Young" <d.s.young.notthisbit@sussex.ac.uk> wrote in message <hscf5g$ktd$1@fred.mathworks.com>...
> First, it's simpler and quicker to use the std function to compute the standard deviations rather than a for loop.
>
> Second, the NaNs in the result are likely to be because the data contain NaNs. So you need to look at how the original matrices are obtained, to see why they contain NaNs. If it makes sense simply to remove NaNs prior to computing the sd, it's easy - there have been quite a number of threads here recently showing how to do it, I think, so I won't go into details, but if you can't find one say so here.

Hi,
I guess my explanation is a bit muddled. The matrices are actually data in .dat files. The loop opens each file, then I use the std function to calculate the standard deviations. All the cells contain data as they are x, y and z coordinates of particles in a fluid flow. And as I said earlier, excel had no problem with the data. I hope this makes things clearer.
Thanks
P.S I used the all function twice on the matrix and it gave a value of 1. I'm assuming this means that all the data are non-zero, real numbers.

CNN wrote:

> I guess my explanation is a bit muddled. The matrices are actually data
> in .dat files. The loop opens each file, then I use the std function to
> calculate the standard deviations. All the cells contain data as they
> are x, y and z coordinates of particles in a fluid flow. And as I said
> earlier, excel had no problem with the data. I hope this makes things
> clearer.
> Thanks
> P.S I used the all function twice on the matrix and it gave a value of
> 1. I'm assuming this means that all the data are non-zero, real numbers.

Is it plausible that you are encountering numeric overflow when evaluating the
sum of squares?

If I recall you indicated that this issue happens for all the data files after
a certain one. Are you checking the return value from fopen() to be sure the
open succeeds? And could you double-check that you fclose() the file once you
are done reading from it? The symptoms are like what I would expect if you are
hitting the limit on the number of simultaneously open files, causing the
fopen() to fail.

Walter Roberson <roberson@hushmail.com> wrote in message <hscp7g$qlv$1@canopus.cc.umanitoba.ca>...
> CNN wrote:
>
> > I guess my explanation is a bit muddled. The matrices are actually data
> > in .dat files. The loop opens each file, then I use the std function to
> > calculate the standard deviations. All the cells contain data as they
> > are x, y and z coordinates of particles in a fluid flow. And as I said
> > earlier, excel had no problem with the data. I hope this makes things
> > clearer.
> > Thanks
> > P.S I used the all function twice on the matrix and it gave a value of
> > 1. I'm assuming this means that all the data are non-zero, real numbers.
>
> Is it plausible that you are encountering numeric overflow when evaluating the
> sum of squares?
>
> If I recall you indicated that this issue happens for all the data files after
> a certain one. Are you checking the return value from fopen() to be sure the
> open succeeds? And could you double-check that you fclose() the file once you
> are done reading from it? The symptoms are like what I would expect if you are
> hitting the limit on the number of simultaneously open files, causing the
> fopen() to fail.

hi,I'm using dlmread to access the files so the issue of opening or closing doesn't arise. And I've used the same code to read more that twice the number of files in a different folder. And I was thinking that since the file gave me the same values when I accessed only one that there's something wrong elsewhere. The code is below

function point = disper83000_5()
    
    point = zeros(1000,3);
    t = 23000:1000:54000;
    n = length(t);
        for i = 1:n;
            x = ['trail',num2str(t(i),'%08.0f'),'.dat'];
            g = dir (x);
            if ~isempty(g)
              M = dlmread(x,'',1,0);
              s = std (M,1);
              point(i,:) = s;
            end
        end
     
     point(n+1:end,:) = [];
     t(all(point==0,2)) = [];
     point(all(point==0,2),:) = [];

For each set of files in different folders, the initial and final values of 't' change but the rest of the code remains the same. I have used it successfully in 3 folders.

Regards.