MATLAB Central Newsreader - import uneven data file

import uneven data file

AMK — Fri, 23 May 2008 13:10:19 -0400

Hi,
I would like to import a comma delimited file that takes the form:

file1:
115,1,2,3,4,5
101,1,2,3,4,5,6,7,8,9

Does a matlab command exist to do this?

ie. convert on-the-fly to this so the workspace can handle it?
115,1,2,3,4,5,nan,nan,nan,nan
101,1,2,3,4,5,6,7,8,9

Then create a variable of just the 115 rows and a variable of just the 101 rows?

x1 = 115,1,2,3,4,5,
x2 = 101,1,2,3,4,5,6,7,8,9

Thanks,
AMK

Re: import uneven data file

Jos — Fri, 23 May 2008 13:28:01 -0400

AMK <kennaster@gmail.com> wrote in message
<7282774.1211548249862.JavaMail.jakarta@nitrogen.mathforum.org>...
> Hi,
> I would like to import a comma delimited file that takes
the form:
>
> file1:
> 115,1,2,3,4,5
> 101,1,2,3,4,5,6,7,8,9
>
> Does a matlab command exist to do this?

D1 = textread('data.txt','%f','delimiter',',')

>
> ie. convert on-the-fly to this so the workspace can handle it?
> 115,1,2,3,4,5,nan,nan,nan,nan
> 101,1,2,3,4,5,6,7,8,9

D2 = load('data.txt')

>
> Then create a variable of just the 115 rows and a variable
of just the 101 rows?
>
> x1 = 115,1,2,3,4,5,
> x2 = 101,1,2,3,4,5,6,7,8,9
>

D1 will not be trivial.
D2 will be relatively easy:

q = isnan(D2) ;
x1 = D2(1,q(1,:))
x2 = ...

hth
Jos

Re: import uneven data file

AMK — Fri, 23 May 2008 17:01:19 -0400

Hi and thank for the reply,

But I'm not clear. For clarification, the data file is large, hundreds of thousands of rows. 101 rows are recorded every minute, and 115 rows are recorded every 15 minutes.

I tried it with just two lines and the D1 command seems to drop everything in a single column. D2 errors our with "??? Error using ==> load
Number of columns on line 1 of ASCII file N:\2008\fiberOpticsAirshed\ml_scripts\finalScripts\data.txt
must be the same as previous lines.", which is my continued problem.

Am I misunderstanding your solution?

Thanks again,
AMK

Re: import uneven data file

AMK — Fri, 23 May 2008 17:04:36 -0400

And just to be clear, the 115 labeled rows have fewer columns than the 101 rows.

AMK

Re: import uneven data file

Andres — Fri, 23 May 2008 20:28:02 -0400

AMK <kennaster@gmail.com> wrote in message
<21277086.1211562307457.JavaMail.jakarta@nitrogen.mathforum.org>...
> And just to be clear, the 115 labeled rows have fewer
columns than the 101 rows.
>
> AMK

So there are two distinct types of rows that you can tell by
there first element? I.e. a rare one beginning with 115
('type A') and the other with 101 ('type B')?! And you know
the number of elements of each row type, or at least that
the type B rows always have N=4 more elements then the type
A ones?!
Then there is a fast way with txt2mat from the file exchange
(sorry I'm biased to it, but I'm the author)

FileOrPath = 'c:\mydata';

A = txt2mat(FileOrPath,'ReplaceExpr',...
{{'115',[repmat('NaN,',1,4) '115']}});

q = isnan(A(:,1)) ;

x1 = A(find(q),5:end);
x2 = A(find(~q),:);

txt2mat can 'natively' handle files with different row
lengths as well (see it's help), but I suppose using the
replacement trick above will speed things up. Feel free to
contact me (via the fex form) if you think the import takes
too long, or if any problems occur.
There are certainly many other solutions.
Best wishes
Andres