Thread Subject:

Creating graphs of data from multiple datafiles

I am trying to create a graph showing multiple datasets of the same type of data in the same format from files currently in a .txt or.dat format. I also have them in excel format. The datasets represent a data value vs time given in the same units in all files.

However the datasets are not all the same length (that is the point...) and the time at which a given data point is recorded is not the same from file to file.

I want to be able to carry out this same procedure for a variable number of files and data types vs time and reckon an m-file is probably the best way to create a regime in which to do this?

It's driving me mad as programming does not come naturally to me! Please help!

So each data file contains two columns? One with time and with your value? If that's the case then it's pretty straightforward. Yes, you need to write a function. The easiest thing is probably to read in the text files. You would do something like this:

D=dir('*.txt'); %find all text files in the current directory

clf %create empty figure window
hold on

for ii=1:length(D) %Loop to process each file in turn
f=fopen(D(ii).name);

%Now you run commands to extract the data; see textscan
%if you have a header, you may need to ignore the first few lines.

fclose(f); %close the text file

plot(timeData,dataValues,'-o')
end
hold off

> However the datasets are not all the same length (that is the point...) and the time at which a given data point is recorded is not the same from file to file.

That doesn't matter at all if you have stored the times in the files. The sort of routine outlined above would just handle that naturally.

Rob,
  You're a ROCK star! Great tip on the use of dir. Do you, or anyone reading this forum, have a suggestion for how to move the data from the files into a new matrix?
For example, I have 3 matrices with the following
file1 = 944x2
file2 = 980x2
file3 = 922x2
and I want to move them all into a new matrix of 980x6. I tried the following
%----code
newfile=zeros(length(file2),6);
newfile(:,1)=file(:,1);
%----
but this wont work because of a dimension mismatch. Whatever! Ideas?

Thanks,
Paul

"Rob Campbell" <matlab@robertREMOVEcampbell.removethis.co.uk> wrote in message <hvasr9$qsa$1@fred.mathworks.com>...
> So each data file contains two columns? One with time and with your value? If that's the case then it's pretty straightforward. Yes, you need to write a function. The easiest thing is probably to read in the text files. You would do something like this:
>
> D=dir('*.txt'); %find all text files in the current directory
>
> clf %create empty figure window
> hold on
>
> for ii=1:length(D) %Loop to process each file in turn
> f=fopen(D(ii).name);
>
> %Now you run commands to extract the data; see textscan
> %if you have a header, you may need to ignore the first few lines.
>
> fclose(f); %close the text file
>
> plot(timeData,dataValues,'-o')
> end
> hold off
>
> > However the datasets are not all the same length (that is the point...) and the time at which a given data point is recorded is not the same from file to file.
>
> That doesn't matter at all if you have stored the times in the files. The sort of routine outlined above would just handle that naturally.

"Paul Hanson" <paul.hanson@mpi.com> wrote in message <i6tf5k$1ft$1@fred.mathworks.com>...
> Rob,
> You're a ROCK star! Great tip on the use of dir. Do you, or anyone reading this forum, have a suggestion for how to move the data from the files into a new matrix?
> For example, I have 3 matrices with the following
> file1 = 944x2
> file2 = 980x2
> file3 = 922x2
> and I want to move them all into a new matrix of 980x6. I tried the following
> %----code
> newfile=zeros(length(file2),6);
> newfile(:,1)=file(:,1);
> %----
> but this wont work because of a dimension mismatch. Whatever! Ideas?
>
>

The error message is telling you that your matrices have different numbers of rows. You would have to do: newfile(1:944,1)=file1(:,1);
However, why are you trying to make a 980x6 matrix? Why not an N-by-2 matrix? Does the same row across files mean the same thing? If so, you obviously have missing data. Why not first make all files have the same number of rows?

> However, why are you trying to make a 980x6 matrix? Why not an N-by-2 matrix? Does the same row across files mean the same thing? If so, you obviously have missing data. Why not first make all files have the same number of rows?

Indeed! The isn't so much as missing as the duration of acquisition was different between the different files. I've been trying to work out how to bring them to the same number of rows but haven't hit upon the right trick yet. *tips welcome!*

A bit more detail: the files are all x,y data but with different lengths. (x is time, y is the dependent variable). What I want to do create a new matrix made from the various files that have had their lengths adjusted, then sort that matrix for a particular signal. The results from that sorting will be moved to yet another matrix that, after some modifications, can be used for principal component analysis. I've got nearly all the other pieces coded, but that adjusting the files to the same number of rows is KILLING me. Is there a truncation command?

In advance, thanks!

Paul

Not one to be a slacker, here's what I've got so far...
---------------
t=dir('*.txt');
tl=zeros(1,length(t));
for i=1:length(t)
    fid=fopen(t(i).name);
    test=textscan(fid,'%f32%f32%f32%f32%f32%f32','Headerlines',3);
    tl(1,i)=length(t{1});
    fclose(fid);
end
tlmin=min(tl); %determines min length for zero matrix
data=zeros(tlmin,2*length(t)); %matrix for the data
for i=1:length(t)
    fid=fopen(traces(i).name); %I miss my days with the NMR/EPR *LOL*
    test=textscan(fid,'%f32%f32%f32%f32%f32%f32','Headerlines',3);
    XYdata(1:tlmin,2*i-1)=test{1};
    XYdata(1:tlmin,2*i)=test{2};
%DENIED! The previous two lines dont work if tlmin<length(test)
end
------------------
I was convinced that'd do the trick. No luck! Ideas?

I think you want to have the same number of rows for each matrix. So you final matrix will have the same number of rows as your smallest file. You'll want to do this for the PCA, I think. As you go through and do the import, store your data in a cell array:
D=dirt('*.txt');
for ii=1:length(D)
dataCell{ii}=myImportDataFunction(D(ii).name);
end

Now you can find the matrix with the smallest number of rows and make your final data set:
minRows=min(cellfun(@length,dataCell));
myFinalMatrix=nan(minRows,length(D)*2);
n=1;
for ii=1:length(D)
 myFinalMatrix(:,n:n+1)=dataCell{ii}(1:minRows,:);
 n=n+2;
end
You can now get your signal from myFinalMatrix(:,2:2:end)
Incidentally, wouldn't all the odd columns be the same since they are time? In that case you'd probably want to make the matrix with only the dependent variable:
myFinalMatrix=nan(minRows,length(D));
for ii=1:length(D)
 myFinalMatrix(:,ii)=dataCell{ii}(1:minRows,2);
end

> Incidentally, wouldn't all the odd columns be the same since they are time? In that case you'd probably want to make the matrix with only the dependent variable:
> myFinalMatrix=nan(minRows,length(D));
> for ii=1:length(D)
> myFinalMatrix(:,ii)=dataCell{ii}(1:minRows,2);
> end

Thanks for the zippy reply! I'll have to dig through the code a bit so I understand it fully, but that'll be for tomorrow. Initially I want to keep the time time increments for each file b/c I have to check that they have the same delta, otherwise I'll have to flippin' normalize everything further. Based upon the lit reference I'm working from, that's probably in the works. I'll update with my results tomorrow! Again, thanks!

cheers and beers,
Paul

Lots of cool tricks in that code of yours! Thanks! I've run into a snag that is primarily a function of understanding arrays. Here's the code:
----
files=dir('*.txt');
for i=1:length(files)
    dcell{i}=textscan(fopen(files(i).name),'%f32%f32%f32%f32%f32%f32','Headerlines',3);
    %fclose(files(i).name);
end
---
This creates the variable dcell that, in this case, is a 1x3 array such that:
>> dcell

dcell =

    {1x6 cell} {1x6 cell} {1x6 cell}
with each element being a 1x6 cell. I want to find the length of the first column of each element in the array. I tried your code of
min(cellfun(@length,dcell))
which returned the value 6, representing the number of columns in each element of the array. I can get the length using
min(cellfun(@length,dcell{1,1}))
and then just iterate through the number of elements with a for loop. Is there a better solution? I haven't found one, yet.

On another note, do you have a good Matlab book reference that illustrates these kind of commands?

As always, thanks!
Paul

> On another note, do you have a good Matlab book reference that illustrates these kind of commands?
>

Sure, just read the Matlab docs. It's all in there and very clearly written.