How to import big data CSV files

11 views (last 30 days)
Hi,
Similar questions have been already asked but I wanted to know if there is an alternative to importing. I have a csv file with 3.5 million rows and 56 columns. At present while importing, I have to select the range I need. For instance I am selecting only 36 columns and barring few rows almost all. I tried textscan() but unable to achieve what I want which is without importing my code should select those rows and columns needed and also my main aim is to save time of importing and since textscan() is pretty fast I was using that. My column values are numerics.
Please let me know if I need to update with something here. I have a 8GB RAM so I think it should be easily feasible.
Thanks in advance!text

Accepted Answer

Walter Roberson
Walter Roberson on 25 May 2019
Edited: per isakson on 25 May 2019
These days, often the most convenient way is to use detectImportOptions, and set the SelectedVariableNames property of that to choose particular columns, and the readtable() -- or as of R2019a, readmatrix() if you are pure numeric.
textscan() is also a possibility. The easiest way to use that might be something like,
numcol = 56;
wanted_cols = [5 17:33 44:47]; %adjust to suit
fmt_cell = repmat('%*s', 1, numcol);
fmt_cell(wanted_cols) = {'%f'};
fmt = [fmt_cell{:}];
data = cell2mat( textscan(fid, fmt, 'Delimiter', ',', 'CollectOutput', 1) );
  17 Comments
Walter Roberson
Walter Roberson on 30 May 2019
Shrug. You can get a wrong and unusable input quickly, or you can get a correct and usable input more slowly.
The extract you showed is not a csv file. csv files are, by definition, Comma Separated Values, never whitespace separated. RFC4180 specifically says that spaces are to be considered part of a field, not a separator.
It is not difficult to extract from a file only the lines that start with numbers. It is not all that much more difficult to divide a file into blocks of numbers, so that non-number marks the end of a block and the blocks are to be imported separately. But as soon as you start wanting to extract particular information from the non-number sections and associating it with a block of numbers, the task becomes more complicated.
Abhishek Singh
Abhishek Singh on 30 May 2019
I am sorry. Importing in matlab showed it in columns and since it was a csv format file I thought it must be like that. But I do understand your point. I think I will stick to the first plan of using textscan() since it was also efficient.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!