MATLAB Answers


How increase bufsize for importdata

Asked by David
on 12 Mar 2011

I am using the importdata function to import data from tab-separated and comma-separated text files. This works great for files up to at least 10Mb, but fails on files with an identical format in the 70Mb range with the following error.

Caused by: Error using ==> textscan Buffer overflow (bufsize = 1000005) while reading string from file (row 1, field 1). Use 'bufsize' option. See HELP TEXTSCAN.

Is there an easy way to increase bufsize directly in importdata call, without mucking around in the textscan function? I understand that as an alternative I can rewrite my code using textscan directly, but my current M-code file is working with importdata for smaller imports and I am looking for the simplest solution to allow import of larger data sets.


Log in to comment.

2 Answers

Answer by Oleg Komarov on 12 Mar 2011
 Accepted Answer

You can try to edit line 319 of importdata:

bufsize = min(1000000, max(numel(fileString),100)) + 5;

Set the minimum threshold 1000000 to something higher.

EDIT 14 March 02:14 GMT 00

You had to specify that "NA" should be treated as empty:

fid = fopen('C:\Users\Oleg\Desktop\ancestry-probs-par2.tsv');
% Column headers
colHead = fgetl(fid); 
colHead = textscan(colHead,'%s');
colHead = colHead{1};  
% get # data columns
numH = length(colHead);
% make fmt
fmt = ['%s', repmat('%f',1,numH)];
  • Import the file in bulk (if enough memory)
% Import file
data = textscan(fid,fmt,'HeaderLines',1,'TreatAsEmpty','NA');
fid = fclose(fid);
  • Import line by line (26 seconds on my pc, preallocation doesn't give the boost since just 191 lines...)
% Import file
data = cell(0,2);
while ~feof(fid)
data = [data; textscan(fid,fmt,1,'HeaderLines',1,'TreatAsEmpty','NA','CollectOutput',1)];
fid = fclose(fid);
rowHead = cat(1,data{:,1});
data = cat(1,data{:,2});



Perfect. This works with a slight modification.
data = cat(2,data{:,2:end});
instead of
data = cat(1,data{:,2});

I forgot to put "'CollectOutput',1" in the bulk import with textscan.

Thanks Oleg this was very helpful. To others if you have a CSV format do not forget that white space is the default delimiter so you need to add 'Delimiter',',' to the textscan arguments. i.e. textscan(fid,fmt,'HeaderLines',1,'Delimiter',',','CollectOutput',1)

Log in to comment.

Answer by Walter Roberson
on 12 Mar 2011

It looks to me as if it is thinking that the first line is more than 1000000 characters.

How long is the first line?

  1 Comment

head -n 1 test_file.tsv | wc -m

So, I tried bufsize = 1612061 + 100
Same error.

Log in to comment.

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Win prizes and improve your MATLAB skills

Play today