fgetl skips lines beginning with #

Hello,

I have a csv file that looks like this:

# dataset: GeoCSV 2.0
# delimiter: ,
# SID: IM_NV01__SHZ
# sample_count: 60001
# sample_rate_hz: 40
# start_time: 2007-07-23T22:30:09.000000Z
# latitude_deg: 38.429501
# longitude_deg: -118.303703
# elevation_m: 2040.0
# depth_m: 0.0
# azimuth_deg: 0.0
# dip_deg: -90.0
# instrument: GS13-NVAR=NV01=Gen=AIM24S_8.11E6=NV01
# scale_factor: 9.2265902E9
# scale_frequency_hz: 1.0
# scale_units: M/S
# field_unit: UTC, M/S
# field_type: datetime, FLOAT
Time, Sample
2007-07-23T22:30:09.000000Z, -9.3528399e-08

The remaing lines are like the last line above (a date/time and data value)

I want to read in all the lines beginning, those with the #, as well as the data lines but I find that using the fgetl command to read a line skips all the lines beginning with #

Here is what I tried as an initial test, but I was surprised by the output as it skipped all the lines beginning with #:

fid = fopen('geocsvmod.csv','r');
tline1 = fgetl(fid);
tline2 = fgetl(fid);
fclose(fid);

The output I got is:

>> tline1

tline1 =

'Time, Sample'

>> tline2

tline2 =

'2007-07-23T22:30:09.000000Z, -9.3528399e-08'

I can't find any documentation that fgetl should skip a line beginning with #

I would like to simply read each line in one by one and process them

as I go, including the lines with #

I hope I am not missing something obvious. Other commands for reading in data such as importdata appear to do this as well

Any advice?

Thank you

[Walter Roberson]

I do not get that when I test, including in that same R2018a release.

[Walter Roberson]

I downloaded your csv file, and constructed the code

fid = fopen('geocsvmod.csv','r');
tline1 = fgetl(fid);
tline2 = fgetl(fid);
fclose(fid);
display(tline1)
display(tline2)

The result I get is

tline1 =

'# dataset: GeoCSV 2.0'

tline2 =

'# delimiter: ,'

[John J. Jasbinsek]

Thank you for the note.

Based on it I had the idea of changing the extension of my data file from .csv to .txt and now it seems to work.

The raw files come to me as .csv but it's easy enough to automate an extension change to .txt, so I think that should work for me now.

Maybe .csv files have some "baggage" about delimiters or something that cause fgetl to do this ...?

[John J. Jasbinsek]

Wow - now I tried it with the .csv extension again, and it works for me now like it did for you.

I did restart Maltab in the meantime but that's really odd.

Anyway - it all seems to work now. Thank you for your time and help.

[Walter Roberson]

I woner if perhaps the actual file was UTF encoded or something like that?

Answer 1

[Image Analyst]

Try this:

% Open the file.
fileID = fopen(fullFileName, 'rt');
% Read the first line of the file.
textLine = fgetl(fileID);
while ischar(textLine)
	% Print out what line we're operating on.
    fprintf('%s\n', textLine);
    if strcmpi(textLine(1), '#')  % You can also use startsWith(textLine, '#') in newer versions.
        % Skip this line
        continue;
    end
    % Process this line if you get here.
    % Now read the next line.
    textLine = fgetl(fileID);
end
% All done reading all lines, so close the file.
fclose(fileID);

[John J. Jasbinsek]

Thank you for the reply.

The solution you provided still skips all the lines at the beginning of the file that begin with a #

The output of:

% Read the first line of the file.

textLine = fgetl(fileID);

is:

textLine =

'Time, Sample'

So this means that fgetl ignored the lines in the csv file at the top starting with #. I think the solution you wrote up is geared towards the opposite of what I would like, i.e. the solution provided looks to ignore the lines beginning with #. The odd thing is, it appears fgetl is doing this on its own.

However, I would like fgetl to not skip the lines beginning with #. I'd like to have each line read from the top. I can't think of why fgetl would automatically skip the lines beginning with #

Thank you

[Image Analyst]

You forgot to attach your file again. I'll create one next year and answer you next year.

[Image Analyst]

OK, it's next year. I've created the file you forgot to attach and ran my code. I'm attaching the file. My code is below:

fullFileName = fullfile(pwd, 'geocsvmod.csv');
% Open the file.
fileID = fopen(fullFileName, 'rt');
% Read the first line of the file.
textLine = fgetl(fileID);
while ischar(textLine)
	% Print out what line we're operating on.
    fprintf('Now processing line %s\n', textLine);
	% Now process this line somehow...
    % Now read the next line.
    textLine = fgetl(fileID);
end
% All done reading all lines, so close the file.
fclose(fileID);

and the output is

Now processing line # dataset: GeoCSV 2.0

Now processing line # delimiter: ,

Now processing line # SID: IM_NV01__SHZ

Now processing line # sample_count: 60001

Now processing line # sample_rate_hz: 40

Now processing line # start_time: 2007-07-23T22:30:09.000000Z

Now processing line # latitude_deg: 38.429501

Now processing line # longitude_deg: -118.303703

Now processing line # elevation_m: 2040.0

Now processing line # depth_m: 0.0

Now processing line # azimuth_deg: 0.0

Now processing line # dip_deg: -90.0

Now processing line # instrument: GS13-NVAR=NV01=Gen=AIM24S_8.11E6=NV01

Now processing line # scale_factor: 9.2265902E9

Now processing line # scale_frequency_hz: 1.0

Now processing line # scale_units: M/S

Now processing line # field_unit: UTC, M/S

Now processing line # field_type: datetime, FLOAT

Now processing line Time, Sample

Now processing line 2007-07-23T22:30:09.000000Z, -9.3528399e-08

You can see that it reads every single line even though the file has the .csv file extension. I'm using R2018b though I don't think it makes a difference. Attach your file and your code, like I did, with the paper clip icon so we can see what you're doing wrong.

[John J. Jasbinsek]

Hello,

Apologies for not attaching the file. Thank you for your analysis. I noted in my response above to another reply that (1) when I changed my file extension to .txt instead of .csv it worked, and then after restarting Matlab, it worked with the original .csv extension too. So the code I was originally working with "worked" - but Matlab was exhibiting a behavior I doubt I will be able to replicate. Thank you for your time and help.

[Image Analyst]

It could be something like you didn't clear all your variables with

clear all 

and when you were using the fileID you were somehow using the one from when it was the other file. Maybe you changed the name of only some fid variables in your program but not all of them or something like that.

[John J. Jasbinsek]

That couldb be it. I generally do a good job of closing "fid" but maybe when I was doing my quick tests I didn't do it. However in spending a little time trying to replicate the problem doing that didn't cause a problem this time. I think I'll never know. Anyway, it seems to always work as expected now.