Importing Column Vectors Sequentially

1 view (last 30 days)
Alasdair Fulton
Alasdair Fulton on 5 Aug 2016
Edited: dpb on 7 Aug 2016
Hi, I'm trying to import data from a large tab delimited CSV with headers. (0.8 GB)
I don't want to import everything, just a number of specific columns. I.e. I would like to create unique column vectors for:
1. Cells D19:D234568
then
2. Cells F19:F234568
and so on.
Currently I'm doing this one-by-one as, even with 12GB ram I'm running out of memory.
There must be a simple way of doing this quickly, no? Once the vectors are loaded and saved as .mat they load up in seconds.
Cheers,
Alasdair
  3 Comments
per isakson
per isakson on 7 Aug 2016
"tab delimited CSV with headers. (0.8 GB) .... even with 12GB ram I'm running out of memory" &nbsp I find it hard to believe that a 0.8GB text-file should cause an out of memory error.
Is it numerical, text or mixed data?
Did you try something like this?
frm = '%*s%*s%*s*f%*s%f%*[^\n]';
cac = textscan( fid, frm, (234568-18), 'Headerlines',18, 'Delimiter',\t')

Sign in to comment.

Accepted Answer

dpb
dpb on 7 Aug 2016
Edited: dpb on 7 Aug 2016
"... in ... import app those are the cells ... tab delimited..."
In that case, use textscan and a format string set up to read the desired columns. This isn't particularly difficult to automate depending on the columns wanted...
cols={'D','F'}; % the list of wanted columns
fmt=[]; % empty string to build format string into
for i=1:length(cols) % over the number of columns to read
fmt=[fmt repmat('%*f',1,cols{i}-'A') '%f']; % skip N-1, read 1
end
fmt=[fmt '%*[^\n']; % and then skip rest of line
fid=fopen('filename','r');
data=cell2mat(textscan(fid,fmt,'delimiter','\t', ...
'headerlines', 18, ...
'collectoutput',1)); % and read the file
fid=fopen(fid);
There's a section Large Text Files linked to at the doc for textscan that describes how to read a file in blocks if this still errors out on memory altho if the import tool can do it, the above likely will work as I'd venture it's what it does as a first try, anyway...

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!