How do I read data (from a .dat file) seperated by lines of text into individual vectors

Hi there,

I am struggling to read in a .dat file correctly. The file has a bunch of headers starting with #s at the beginning followed by data (2 columns) followed by another header and then more data etc. The data I wish to read into matlab is over 2 million lines so you can imagine that having to go back and delete these headers to load chunks of data in individually will take some time.

Is there a way for Matlab to read in a file like this and seperate the data into seperate vectors each time there's a line of text/ a gap (if I have to go through and delete all the headers). It will not 'load' the data as it contains text that over runs the amount of columns of data at the start. I have tried using the import tool in matlab and I have tried fscanf but not had much luck either.

Any advice would be much much appreciated.

Many thanks

Answer 1

[Walter Roberson]

S = regexp(fileread('YourFileName.dat'), '\r?\n', 'split');
if isempty(S{end}); S(end) = []; end    %regexp split leaves empty at bottom if file ended in \n which is common
nonheader = cellfun(@isempty, regexp(S, '^\s*#'));  %permit space before #
starts = strfind([false nonheader], [false true]);
stops = strfind([nonheader false], [true false]);
num_blocks = length(starts);
blocks = cell(num_blocks, 1);
for K = 1 : num_blocks
    fields = regexp(S(starts(K):stops(K)), '\s+', 'split');
    lens = cellfun(@length, fields);
    if any(lens ~= 2)
        error('data that does not have exactly two fields is present between lines %d and %d', starts(K), stops(K));
    end
    temp = vertcat(fields{:});
    blocks{K} = str2double(temp);
    end
end

Now blocks will be a cell array consisting of numeric arrays with two columns each.

[P_L]

Hi Walter, thank you so much for getting back to me!!! I now have all my data how I wanted. I believe it all stored in the blocks variable (which contains 14685 x 1 cell) that then seperates into the correct bunches of data.

If I now wanted to plot this data as x, y is it possible to do so? How would I reference them? I am trying this but not sure how to refrence the cells that within the blocks {1,1}. I've never plotted anything like that before.

Many many thanks

x= blocks(1(:,1)) % for the first row and the bunch of data stored in it
y = blocks(2(:,2) % for the second row and the bunch of data stored in it
plot(x,y)?

[Walter Roberson]

hold on
cellfun(@(B) plot(B(:,1), B(:,2)), blocks);
hold off

You can omit the hold commands if hold is already in effect. However, you need hold to be in effect when the second plot() is executed, or else each plot() will end up removing the previous lines.

You could potentially have problems with xlim and ylim not ending up set nicely.

If you want to loop through the blocks then,

for K = 1 : length(blocks)
    x = blocks{K}(:,1);
    y = blocks{K}(:,2);
    ..... do things
end

[P_L]

Hi Walter, thank you so much. You are a genius!!!

I have used the 1st suggestion you gave me after trying both. This is because the first attempt runs much more quickly as you can imagine. I have also managed to secure the axis using axis([xmin xmax ymin ymax]).

I am now trying to set the colour of the plots as black by using both 'k-' and 'MarkerColor', 'k'. Both of these attempts don't seem to work. I have also seen online that you could use:

set(h(1),'MarkerEdgeColor','r','MarkerFaceColor','r')

as my plot has around 1500 "blcoks", is that what I need to reference in this command? e.g.

set(h(1:1500),'-','MarkerFaceColor','k') but that too has also been unsuccessful.

Please see attached. I would just like my lines to be a uniform colour.

I appreciate all your help so far. If you would be so kind to help with my last request, that really would be amazing.

Many thanks