Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Reading textfile line by line and storing into an array

Subject: Reading textfile line by line and storing into an array

From: Jane

Date: 11 Jan, 2011 17:30:25

Message: 1 of 3

Hi,

I have a textfile that contains many different types of data. The first 6 characters of each line determines what type of data is stored in that line. So I want to be able to read each line of the textfile, check if the first 6 characters contain the string identifying the data I need and if so store the information to an array. So far I have

[filename, pathname] = uigetfile('*.gvl');
fid = fopen([pathname,filename]);

tline = fgetl(fid);

i = 0;
while ischar(tline)
    if tline(1:6) == '$GF3DW'
        i = i + 1;
        C{i} = textscan(tline,'%s%n%n%n%n%s','delimiter',',');
    end
    tline = fgetl(fid);
end
fclose(fid)

But C turns out to be pretty monsterous, A (1, 40321) cell array with each cell being (1,6) in size. Is there a better way of storing the information?

Example data:

$GF3DW, 2.44, 244.2, -6.5, 1.62, 2008/03/12 09:17:02*23
$GF3DW, 2.47, 244.7, -4.1, 1.62, 2008/03/12 09:17:02*23
$GFDTA,1,0.0,0,1.0,15,2008/03/12 09:17:02, CH4OP-1023,2830*57
$GF3DW, 2.44, 246.3, -4.1, 1.63, 2008/03/12 09:17:02*27
$GF3DW, 2.41, 247.6, -3.9, 1.63, 2008/03/12 09:17:02*29
$GF3DW, 2.24, 248.5, -3.2, 1.59, 2008/03/12 09:17:02*24
$GF3DW, 2.17, 245.2, -0.7, 1.61, 2008/03/12 09:17:02*23
$GF3DW, 2.16, 245.8, 2.9, 1.59, 2008/03/12 09:17:03*03
$GF3DW, 2.05, 250.5, 0.8, 1.59, 2008/03/12 09:17:03*0B
$GFDTA,1,0.0,0,1.0,18,2008/03/12 09:17:03, CH4OP-1023,2830*5B
$GF3DW, 2.11, 254.3, -0.7, 1.62, 2008/03/12 09:17:03*26
$GF3DW, 2.20, 257.0, -0.3, 1.64, 2008/03/12 09:17:03*26
$GF3DW, 2.12, 251.8, -2.2, 1.62, 2008/03/12 09:17:03*2C
$GF3DW, 2.14, 253.6, -3.2, 1.64, 2008/03/12 09:17:03*21
$GF3DW, 2.11, 255.8, -1.6, 1.62, 2008/03/12 09:17:03*2C
$GF3DW, 2.00, 259.2, -0.7, 1.62, 2008/03/12 09:17:03*2A
$GF3DW, 1.98, 263.2, -1.2, 1.59, 2008/03/12 09:17:04*2A
$GF3DW, 1.89, 261.6, 0.5, 1.62, 2008/03/12 09:17:04*0F

I would appreciate any help
Many thanks
Jane

Subject: Reading textfile line by line and storing into an array

From: Donn Shull

Date: 11 Jan, 2011 18:17:05

Message: 2 of 3

"Jane " <j.l.terry@hotmail.co.uk> wrote in message <igi43h$rcj$1@fred.mathworks.com>...
> Hi,
>
> I have a textfile that contains many different types of data. The first 6 characters of each line determines what type of data is stored in that line. So I want to be able to read each line of the textfile, check if the first 6 characters contain the string identifying the data I need and if so store the information to an array. So far I have
>
> [filename, pathname] = uigetfile('*.gvl');
> fid = fopen([pathname,filename]);
>
> tline = fgetl(fid);
>
> i = 0;
> while ischar(tline)
> if tline(1:6) == '$GF3DW'
> i = i + 1;
> C{i} = textscan(tline,'%s%n%n%n%n%s','delimiter',',');
> end
> tline = fgetl(fid);
> end
> fclose(fid)
>
> But C turns out to be pretty monsterous, A (1, 40321) cell array with each cell being (1,6) in size. Is there a better way of storing the information?
>
> Example data:
>
> $GF3DW, 2.44, 244.2, -6.5, 1.62, 2008/03/12 09:17:02*23
> $GF3DW, 2.47, 244.7, -4.1, 1.62, 2008/03/12 09:17:02*23
> $GFDTA,1,0.0,0,1.0,15,2008/03/12 09:17:02, CH4OP-1023,2830*57
> $GF3DW, 2.44, 246.3, -4.1, 1.63, 2008/03/12 09:17:02*27
> $GF3DW, 2.41, 247.6, -3.9, 1.63, 2008/03/12 09:17:02*29
> $GF3DW, 2.24, 248.5, -3.2, 1.59, 2008/03/12 09:17:02*24
> $GF3DW, 2.17, 245.2, -0.7, 1.61, 2008/03/12 09:17:02*23
> $GF3DW, 2.16, 245.8, 2.9, 1.59, 2008/03/12 09:17:03*03
> $GF3DW, 2.05, 250.5, 0.8, 1.59, 2008/03/12 09:17:03*0B
> $GFDTA,1,0.0,0,1.0,18,2008/03/12 09:17:03, CH4OP-1023,2830*5B
> $GF3DW, 2.11, 254.3, -0.7, 1.62, 2008/03/12 09:17:03*26
> $GF3DW, 2.20, 257.0, -0.3, 1.64, 2008/03/12 09:17:03*26
> $GF3DW, 2.12, 251.8, -2.2, 1.62, 2008/03/12 09:17:03*2C
> $GF3DW, 2.14, 253.6, -3.2, 1.64, 2008/03/12 09:17:03*21
> $GF3DW, 2.11, 255.8, -1.6, 1.62, 2008/03/12 09:17:03*2C
> $GF3DW, 2.00, 259.2, -0.7, 1.62, 2008/03/12 09:17:03*2A
> $GF3DW, 1.98, 263.2, -1.2, 1.59, 2008/03/12 09:17:04*2A
> $GF3DW, 1.89, 261.6, 0.5, 1.62, 2008/03/12 09:17:04*0F
>
> I would appreciate any help
> Many thanks
> Jane

Hi Jane,

You probably would be better off using a numeric array. You could drop the '$GF3DW' since is is redundant and convert the date string to a number using datenum. for example if you do not need the resolution provided by the last two digits then:

temp = '2008/03/12 09:17:02*23'
serialDateNumber = datenum(temp(end-3) , 'yyyy/mm/dd HH:MM:SS')

then you could store the information in a (5, 40321) numeric array. You can also use end addressing in building your array so you don't need the index i. That is each item is added to the array with:

C{end+1} =

Your example and this method both creat an array which grows within a loop which is wasteful in terms ofmemory usage and will be slower than if you are able to determine the size of your result ahead of time and preallocate the array before entering the loop and then just fill the array with data in the loop.

Hope this helps,

Donn

Subject: Reading textfile line by line and storing into an array

From: Jane

Date: 11 Jan, 2011 18:45:05

Message: 3 of 3

"Donn Shull" <donn.shull.no_spam@aetoolbox.com> wrote in message <igi6r1$n8q$1@fred.mathworks.com>...
> "Jane " <j.l.terry@hotmail.co.uk> wrote in message <igi43h$rcj$1@fred.mathworks.com>...
> > Hi,
> >
> > I have a textfile that contains many different types of data. The first 6 characters of each line determines what type of data is stored in that line. So I want to be able to read each line of the textfile, check if the first 6 characters contain the string identifying the data I need and if so store the information to an array. So far I have
> >
> > [filename, pathname] = uigetfile('*.gvl');
> > fid = fopen([pathname,filename]);
> >
> > tline = fgetl(fid);
> >
> > i = 0;
> > while ischar(tline)
> > if tline(1:6) == '$GF3DW'
> > i = i + 1;
> > C{i} = textscan(tline,'%s%n%n%n%n%s','delimiter',',');
> > end
> > tline = fgetl(fid);
> > end
> > fclose(fid)
> >
> > But C turns out to be pretty monsterous, A (1, 40321) cell array with each cell being (1,6) in size. Is there a better way of storing the information?
> >
> > Example data:
> >
> > $GF3DW, 2.44, 244.2, -6.5, 1.62, 2008/03/12 09:17:02*23
> > $GF3DW, 2.47, 244.7, -4.1, 1.62, 2008/03/12 09:17:02*23
> > $GFDTA,1,0.0,0,1.0,15,2008/03/12 09:17:02, CH4OP-1023,2830*57
> > $GF3DW, 2.44, 246.3, -4.1, 1.63, 2008/03/12 09:17:02*27
> > $GF3DW, 2.41, 247.6, -3.9, 1.63, 2008/03/12 09:17:02*29
> > $GF3DW, 2.24, 248.5, -3.2, 1.59, 2008/03/12 09:17:02*24
> > $GF3DW, 2.17, 245.2, -0.7, 1.61, 2008/03/12 09:17:02*23
> > $GF3DW, 2.16, 245.8, 2.9, 1.59, 2008/03/12 09:17:03*03
> > $GF3DW, 2.05, 250.5, 0.8, 1.59, 2008/03/12 09:17:03*0B
> > $GFDTA,1,0.0,0,1.0,18,2008/03/12 09:17:03, CH4OP-1023,2830*5B
> > $GF3DW, 2.11, 254.3, -0.7, 1.62, 2008/03/12 09:17:03*26
> > $GF3DW, 2.20, 257.0, -0.3, 1.64, 2008/03/12 09:17:03*26
> > $GF3DW, 2.12, 251.8, -2.2, 1.62, 2008/03/12 09:17:03*2C
> > $GF3DW, 2.14, 253.6, -3.2, 1.64, 2008/03/12 09:17:03*21
> > $GF3DW, 2.11, 255.8, -1.6, 1.62, 2008/03/12 09:17:03*2C
> > $GF3DW, 2.00, 259.2, -0.7, 1.62, 2008/03/12 09:17:03*2A
> > $GF3DW, 1.98, 263.2, -1.2, 1.59, 2008/03/12 09:17:04*2A
> > $GF3DW, 1.89, 261.6, 0.5, 1.62, 2008/03/12 09:17:04*0F
> >
> > I would appreciate any help
> > Many thanks
> > Jane
>
> Hi Jane,
>
> You probably would be better off using a numeric array. You could drop the '$GF3DW' since is is redundant and convert the date string to a number using datenum. for example if you do not need the resolution provided by the last two digits then:
>
> temp = '2008/03/12 09:17:02*23'
> serialDateNumber = datenum(temp(end-3) , 'yyyy/mm/dd HH:MM:SS')
>
> then you could store the information in a (5, 40321) numeric array. You can also use end addressing in building your array so you don't need the index i. That is each item is added to the array with:
>
> C{end+1} =
>
> Your example and this method both creat an array which grows within a loop which is wasteful in terms ofmemory usage and will be slower than if you are able to determine the size of your result ahead of time and preallocate the array before entering the loop and then just fill the array with data in the loop.
>
> Hope this helps,
>
> Donn

Donn,

Thanks for you help. I now have the following code

[filename, pathname] = uigetfile('*.gvl');
fid = fopen([pathname,filename]);

tline = fgetl(fid);

C = [];

while ischar(tline)
    if tline(1:6) == '$GF3DW'
        temp = textscan(tline,'%s%n%n%n%n%s','delimiter',',');
        C(end+1,:) = [temp{2:5}, datenum(temp{6}{1}(1:end-3),'yyyy/mm/dd HH:MM:SS')];
    end
    tline = fgetl(fid);
end
fclose(fid);

which works a treat, even if as you say it's very slow to execute with C increasing within the loop. However, I only need to run it once and can then save the array to a .mat file for future use, so I'm not overly concerned by that.

Thanks again for you help
Jane

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us