Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Using TEXTSCAN to import an ASCII file with a header and blank lines between different data sets

Asked by Kristia on 27 Mar 2013

I have several text files that represent a house and each file has several data sets that represent a room within the house.

The text file looks similar to the following but a majority of the data has been deleted. Each zone has 1440 lines of data and each house has a different number of zones:

 project:  House1_1                              Tue Mar 19 12:30:42 2013
 description:  
 date   time    time       Ozone
       of day    [s]     [kg/kg]
 level: firstfloor    zone: bedroom1
 Jan01 00:00:00     0  0.000e+000
 Jan01 00:01:00    60  1.487e-009
 Jan01 00:02:00   120  5.330e-009
 Jan01 00:03:00   180  1.084e-008
 Jan01 23:57:00 86220  1.575e-007
 Jan01 23:58:00 86280  1.575e-007
 Jan01 23:59:00 86340  1.575e-007
 Jan01 24:00:00 86400  1.575e-007
 level: firstfloor    zone: kitchen
 Jan01 00:00:00     0  0.000e+000
 Jan01 00:01:00    60  1.483e-009
 Jan01 00:02:00   120  5.315e-009
 Jan01 00:03:00   180  1.081e-008
 Jan01 23:57:00 86220  1.564e-007
 Jan01 23:58:00 86280  1.564e-007
 Jan01 23:59:00 86340  1.564e-007
 Jan01 24:00:00 86400  1.564e-007
 level: firstfloor    zone: bedroom2
 Jan01 00:00:00     0  0.000e+000
 Jan01 00:01:00    60  1.486e-009
 Jan01 00:02:00   120  5.321e-009
 Jan01 00:03:00   180  1.081e-008
 Jan01 23:57:00 86220  1.549e-007
 Jan01 23:58:00 86280  1.549e-007
 Jan01 23:59:00 86340  1.549e-007
 Jan01 24:00:00 86400  1.549e-007

The final goal is to generate a graph of ozone concentration versus time for each house that contains all of the zones for that house. Presently I am having trouble importing the data. I can use the following code to open the first zone in one file. I only need the data from the fourth column. I do not need the first 9 lines (header info) or the 3 lines in between zones but I need the data for each zone to be its own data set.

fid=fopen('House1-1.txt');
temp=textscan(fid,'%*s %*s %*d %f','Headerlines',9);
fclose(fid);  

I can not figure out how to create a loop to read to the end of each file and get the data for each zone into its own array. I also need the loop to read each house file within the folder. Any help would be appreciated.

0 Comments

Kristia

Products

No products are associated with this question.

3 Answers

Answer by Cedric Wannaz on 27 Mar 2013
Edited by Cedric Wannaz on 27 Mar 2013
Accepted answer

An alternative could be to use REGEXP to get blocks of data, e.g. in a struct array, and then post-process the content. To illustrate using the content that you gave:

 >> buffer  = fileread('myData.txt') ;
 >> pattern = 'level:\s*(?<level>\S+)\s+zone:\s*(?<zone>\S+)\s*(?<data>.*?)(?=($|level))' ;
 >> blocks = regexp(buffer, pattern, 'names' )
 blocks = 
 1x3 struct array with fields:
    level
    zone
    data
 >> blocks(1)
 ans = 
    level: 'firstfloor'
     zone: 'bedroom1'
     data: [1x282 char]
 >> blocks(2)
 ans = 
    level: 'firstfloor'
     zone: 'kitchen'
     data: [1x282 char]
 >> blocks(3)
 ans = 
    level: 'firstfloor'
     zone: 'bedroom2'
     data: [1x277 char]

So, using a simple loop, you can process all blocks already parsed:

 for k = 1 : length(blocks)
    fprintf('Level = %s, zone = %s\n', blocks(k).level, blocks(k).zone) ;
    ... do something, e.g. with textscan, on blocks(k).data
 end

0 Comments

Cedric Wannaz
Answer by per isakson on 27 Mar 2013
Edited by per isakson on 27 Mar 2013

Here is one of many alternate solutions.

    >> [ header, block_head, block_data ] = cssm()
    header = 
        ' project:  House1_1                              Tue Mar 19 12:30:42 2013'
        ''
        ' description:  '
        ''
        ' date   time    time       Ozone'
        '       of day    [s]     [kg/kg]'
        ''
    block_head = 
        ' level: firstfloor    zone: bedroom1'
        'zone: kitchen'
        'zone: bedroom2'
    block_data = 
        [8x1 double]
        [8x1 double]
        [8x1 double]
    >> 

The values of block_head are obviously corrupted.

where cssm is

    function [ header, block_head, block_data ] = cssm()
    %%    
        fid = fopen( 'cssm.txt' );
    %   cac = textscan( fid, '%[^\n]' ); swallows empty lines
        cac = textscan( fid, '%s', 'Delimiter', '\n' );
        fclose( fid );
    %%    
        ixs = find( strncmp( 'level:', cac{:}, 6 ) );
    %%
        fid = fopen( 'cssm.txt' );
        header = cell( ixs(1)-1, 1 ); 
        for ii = 1 : ixs(1)-1
            header{ii} = fgetl( fid );
        end
    %%
        nnblock     = numel( ixs );
        ixs(end+1)  = size( cac{:}, 1 );
        block_head  = cell( nnblock, 1 );
        block_data  = cell( nnblock, 1 );
        for iib = 1 : nnblock
            block_head{iib} = fgetl( fid );
            block_data(iib) = textscan(fid,'%*s%*s%*d%f', ixs(iib+1)-ixs(iib) );   
        end
        fclose( fid );
    end

and cssm.txt consist of the data line in your question.

.

Next try without reading block_head:

    >> [ header, block_head, block_data ] = cssm()
    header = 
        ' project:  House1_1                              Tue Mar 19 12:30:42 2013'
        ''
        ' description:  '
        ''
        ' date   time    time       Ozone'
        '       of day    [s]     [kg/kg]'
    block_head = 
        []
        []
        []
    block_data = 
        [8x1 double]
        [8x1 double]
        [8x1 double]

where cssm is

    function [ header, block_head, block_data ] = cssm()
    %%    
        fid = fopen( 'cssm.txt' );
    %   cac = textscan( fid, '%[^\n]' ); swallows empty lines
        cac = textscan( fid, '%s', 'Delimiter', '\n' );
        fclose( fid );
    %%    
        ixs = find( strncmp( 'level:', cac{:}, 6 ) );
    %%
        fid = fopen( 'cssm.txt' );
        header = cell( ixs(1)-2, 1 ); 
        for ii = 1 : ixs(1)-2
            header{ii} = fgetl( fid );
        end
    %%
        nnblock     = numel( ixs );
        ixs(end+1)  = size( cac{:}, 1 ) + 2;
        block_head  = cell( nnblock, 1 );
        block_data  = cell( nnblock, 1 );
        for iib = 1 : nnblock
            block_data(iib) = textscan( fid, '%*s%*s%*d%f'      ...
                                    ,   ixs(iib+1)-ixs(iib)-3   ...
                                    ,   'Headerlines', 3        );   
        end
        fclose( fid );
    end

.

One more try:

    >> [ header, block_head, block_data ] = cssm()
    header = 
        ' project:  House1_1                              Tue Mar 19 12:30:42 2013'
        ''
        ' description:  '
        ''
        ' date   time    time       Ozone'
        '       of day    [s]     [kg/kg]'
    block_head = 
        {3x1 cell}
        {3x1 cell}
        {3x1 cell}
    block_data = 
        [8x1 double]
        [8x1 double]
        [8x1 double]
    >> block_head{1}
    ans = 
        ''
        'level: firstfloor    zone: bedroom1'
        ''
    >> block_head{2}
    ans = 
        ''
        ''
        'level: firstfloor    zone: kitchen'
    >> block_head{3}
    ans = 
        ''
        ''
        'level: firstfloor    zone: bedroom2'

block_head contains two successive empty "lines" in block_head 2 and 3. However, the data file does nowhere display an empty line after another empty line. I find this strange.

where

    function [ header, block_head, block_data ] = cssm()
    %%    
        fid = fopen( 'cssm.txt' );
    %   cac = textscan( fid, '%[^\n]' ); swallows empty lines
        cac = textscan( fid, '%s', 'Delimiter', '\n' );
        fclose( fid );
    %%    
        ixs = find( strncmp( 'level:', cac{:}, 6 ) );
    %%
        fid = fopen( 'cssm.txt' );
        header = cell( ixs(1)-2, 1 ); 
        for ii = 1 : ixs(1)-2
            header{ii} = fgetl( fid );
        end
    %%
        nnblock     = numel( ixs );
        ixs(end+1)  = size( cac{:}, 1 ) + 2;
        block_head  = cell( nnblock, 1 );
        block_data  = cell( nnblock, 1 );
        for iib = 1 : nnblock
            block_head(iib) = textscan( fid, '%s', 3, 'Delimiter', '\n' ); 
            block_data(iib) = textscan( fid, '%*s%*s%*d%f'      ...
                                    ,   ixs(iib+1)-ixs(iib)-3   ...
                                    ,   'Headerlines', 0        );   
        end
        fclose( fid );
    end

.

Discussion:

There must be a better way to handle empty lines.

0 Comments

per isakson
Answer by Kristia on 27 Mar 2013

Thanks for the suggestions but I am really confused by both of the answers posted so far. The first answer from Per Isakson keeps giving me error messages such as unexpected MATLAB operator or Function definitions are not permitted in this context and the second answer from Cedric Wannaz gives me the unexpected MATLAB operator error as well. All of the errors are occurring from the first line of code.

I am really new to MATLAB so I am not completely sure what either of these codes are doing so I really don't know what I am doing wrong when I try to put it into my MATLAB but I can't get past the first line of either suggestion.

I did find that I can also use dlmread to import the first data set but again I don't know how to get to the rest of the data using a loop. The following is what I did using dlmread:

M=dlmread('House1-1.txt','',[9 3 1449 3]);

Doing this gives me all of the column 4 data for zone 1. I am not sure if there is a way to loop this and tell it to skip the next 3 rows and then import the next 1440 rows to give me the data for zone 2 and repeat again until the end of the file but something like this is what I need. I only need the 4th column of data (the ozone concentration). I do not need any of the header information imported.

From everything that I have read it seems dlmread should not work at all and that I need to use textscan but I got the above line to import the data. I was also able to change the range to get the 4th column data for the other zones as well but I can't put it into a loop.

Thanks for the help!

6 Comments

Cedric Wannaz on 29 Mar 2013

You're welcome. If you copy and paste the code that I wrote in my EDIT above in an M-file, it should be working directly as it is (if the M-file is saved in the same directory as the file House1-1.txt).

Kristia on 1 Apr 2013

I did copy your edit and it works great! Thanks again!!

Cedric Wannaz on 1 Apr 2013

You're welcome! Don't forget to [ Accept ] one of the answers if it helped, and if you accept mine, don't forget to /\ vote for Per Isakson's answer as well, because he took time to write and test a quite complete answer that is indeed the standard way for processing this kind of file structure (my answer is more compact, but less standard).

Kristia

Contact us