Reading data from file with breaks

9 views (last 30 days)
Timothy Devenport
Timothy Devenport on 21 May 2015
Commented: Star Strider on 22 May 2015
I'm trying to read values from some very large files and store them as a matrix. The issues is as well as having headers at the top, there are also headers repeated every 10k lines or so. For example
header 1, header 2, header 3;
1 2 3;
%(9998 rows of numbers)
4 5 6;
header 1, header 2, header 3;
7 8 9
dlmread doesn't work, and I've tried textscan and fread, which both sort of do what I want but not quite. Thanks in advance.
  2 Comments
Thorsten
Thorsten on 21 May 2015
It is easier to answer such questions if you provide a sample file.
Timothy Devenport
Timothy Devenport on 21 May 2015
Edited: Timothy Devenport on 22 May 2015
Sample uploaded

Sign in to comment.

Answers (1)

Star Strider
Star Strider on 21 May 2015
The textscan function will work, but you have to use fseek to restart it each time it hits the interim text line every 10K or so lines.
This is an archived code snippet to illustrate the technique:
fidi = fopen('S21.txt');
D1 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D1{:}([1:5 end-4:end],:) % Diagnostic Look
fseek(fidi,0,0); % Position Start Of Second Part Of File
D2 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D2{:}([1:5 end-4:end],:) % Diagnostic Look
The ‘Diagnostic Look’ lines verify that the file is importing correctly. They are not important for the code and you can delete them.
  6 Comments
Timothy Devenport
Timothy Devenport on 22 May 2015
This appears to be on the right lines, but I'm not sure it's working quite right. It seems to cut down the data files to just 30 cells in D1 and D2, and that's clearly incorrect, as there should be hundreds of cells.
I'm wondering if it's skipping the header lines, reading a line, then skipping the next lines (as if they're headers) and reading a line and so on...
Star Strider
Star Strider on 22 May 2015
The ‘D1’ and ‘D3’ lines are there only to verify that the code is reading the file correctly. They display the first 5 and last 5 lines in each segment to be sure they were imported correctly. They are otherwise not part of the actual code, so you can delete them.
When I run the code I posted, then add these two lines:
SizeD1 = size(D{1}{1})
SizeD2 = size(D{2}{1})
I get:
SizeD1 =
200 3
SizeD2 =
335 3
I didn’t count the lines manually, but it seems to be working as it should, at least with the file you provided.
While I’m thinking about it, this is the way to concatenate them if you want to:
Dcat = [D{1}{1}; D{2}{1}];
producing a (535x3) double array with the file you provided.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!