MATLAB Central Newsreader - Problems with importing large text files

Problems with importing large text files

Sven B? — Wed, 17 Dec 2008 10:15:33 -0500

Hi,
from my PSPICE-simulation I get a text-file with the simulated data.
The text file is approx. 500MB.
The data in the file consists of 101 columns and approx. 100000 rows.
For Matlab it is not possible to import this file via the Import Wizard.
The first row is normal text, the others should be numbers.
The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
Each column is the result of a Monte Carlo analysis.

The Import wizard starts up an then hangs on the point "creating preview".

Does anyone can help me?

Regards,

Sven

Re: Problems with importing large text files

Sebastiaan — Wed, 17 Dec 2008 10:50:19 -0500

You can write your own import function using fopen:

Assuming your data looks something like this:
Comments
2.35 93.1 294.1 593.1
23.1 345.4 127812947.12834 1282881.1

Then:
A = zeros(100000, 101); % This is ~ 800MB, are you sure you have enough memory?

fid = fopen('data.txt', 'r');
row = 1;

% Skip first line
tline = fgetl(fid);

while 1
  tline = fgetl(fid);
  if ~ischar(tline), break, end
  A(row, :) = str2double(regexp(tline, ' ', 'split'));
end
fclose(fid);

If you have binary data, use fread instead.

Good luck.

"Sven B?" <sven.boehm@contiautomotive.com> wrote in message <giajg4$f3t$1@fred.mathworks.com>...
> Hi,
> from my PSPICE-simulation I get a text-file with the simulated data.
> The text file is approx. 500MB.
> The data in the file consists of 101 columns and approx. 100000 rows.
> For Matlab it is not possible to import this file via the Import Wizard.
> The first row is normal text, the others should be numbers.
> The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
> Each column is the result of a Monte Carlo analysis.
>
> The Import wizard starts up an then hangs on the point "creating preview".
>
> Does anyone can help me?
>
> Regards,
>
> Sven

Re: Problems with importing large text files

Andres — Wed, 17 Dec 2008 11:06:02 -0500

"Sven B?" <sven.boehm@contiautomotive.com> wrote in message <giajg4$f3t$1@fred.mathworks.com>...
> Hi,
> from my PSPICE-simulation I get a text-file with the simulated data.
> The text file is approx. 500MB.
> The data in the file consists of 101 columns and approx. 100000 rows.
> For Matlab it is not possible to import this file via the Import Wizard.
> The first row is normal text, the others should be numbers.
> The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
> Each column is the result of a Monte Carlo analysis.
>
> The Import wizard starts up an then hangs on the point "creating preview".

Seems like the import wizard is overchallenged... but i guess you can go without it. You obviously know everthing about the the file layout, so a textscan (+ fgetl) should do, like

nCol = 101;
fid = fopen('myfile.txt');
firstRowString = fgetl(fid);
data = textscan(fid, repmat('%f',1,nCol),'CollectOutput', true);
fclose(fid)

Maybe you have to adjust for the delimiters in the data section or other details, see the textscan doc. I *assume* you'll get no memory issues in this way.

Re: Problems with importing large text files

Sven B? — Wed, 17 Dec 2008 11:59:04 -0500

Thanks to both of you.
The one from Andres was the thing that saved my day....

Sven