Clear Filters
Clear Filters

Cannot import/load/read complete .dat file

6 views (last 30 days)
B.kun
B.kun on 1 Dec 2014
Edited: Stephen23 on 1 Dec 2014
Hi
I am trying to open my .dat file that has 785663 rows x 29 columns; each column separated by a comma (therefore, csv). I tried using textscan, importdata, csvread, load, etc in order to access my files and hence do some data analysis. The first 2 lines of my .dat file (excluding the headerline) look like this:
"2013-01-02 21:10:00",64359,18.33,20.41257,19.95976,11.60755,4.172891,56.92463,35.29119,10.93861,0,0,0,0,2.630428,0.4060569,0.7230025,0.3542804,-0.0027175,0.2566106,2.75231,2.727983,114.6311,15.36885,7.615118,99.7831,0,23.01129,12.45
"2013-01-02 21:15:00",64360,18.27,20.36502,19.92328,11.72347,4.227652,57.53248,35.50731,10.93866,0,0,0,0,2.427403,0.3657977,0.7309517,0.3855992,0.03224583,0.2124791,2.561898,2.535069,113.2416,16.75837,8.289121,99.78176,0,22.88334,12.47
The first column is date and the rest is all numeric.
I tried using the following code:
fid = fopen(filename,'r');
formatspec = ['%s' repmat('%f',1,29)];
A = textscan(fid, formatspec, 'HeaderLines',4,'Delimiter',',' ,inf, 'CollectOutput', true);
But my A cell array returns me only 11184 rows in some columns, and 11183 rows in other columns.
I also tried importdata, and again, it gave me only 11184 rows for all the columns which are numeric except the date column (11188 rows: because include the headerlines which are text).
In anyway, I couldn't get all my 785663 rows loaded. How can I get the remaining part of my dataset?
Thank you.
  1 Comment
Stephen23
Stephen23 on 1 Dec 2014
Edited: Stephen23 on 1 Dec 2014
Did you look at rows 11184/11185? Is it possible that the data on these rows might be incomplete or not able to be parsed with either of the format specifiers that you used?
When I run the code I get an error "??? Error using ==> textscan Param/value pairs must come in pairs." This seems to come from the inf value which is entered as one of the textscan options. Removing this inf gives me a 1x2 cell... but with empty cells, due to not having header lines :)
I made some small changes to the code you supplied (particularly the string format), and this imports the data correctly:
filename = 'temp.txt';
fid = fopen(filename,'r');
formatspec = ['%q',repmat('%f',1,28)];
A = textscan(fid, formatspec, 'Delimiter',',', 'CollectOutput', true);
fclose(fid);
It is likely to be something specific to the rows where these formats do not parse the data...
Note that using importdata also works for me:
B = importdata(filename, ',');

Sign in to comment.

Answers (0)

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!