Got Questions? Get Answers.
Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Problems with importing large text files

Subject: Problems with importing large text files

From: Sven B?

Date: 17 Dec, 2008 10:15:33

Message: 1 of 4

Hi,
from my PSPICE-simulation I get a text-file with the simulated data.
The text file is approx. 500MB.
The data in the file consists of 101 columns and approx. 100000 rows.
For Matlab it is not possible to import this file via the Import Wizard.
The first row is normal text, the others should be numbers.
The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
Each column is the result of a Monte Carlo analysis.

The Import wizard starts up an then hangs on the point "creating preview".

Does anyone can help me?

Regards,

Sven

Subject: Problems with importing large text files

From: Sebastiaan

Date: 17 Dec, 2008 10:50:19

Message: 2 of 4

You can write your own import function using fopen:

Assuming your data looks something like this:
Comments
2.35 93.1 294.1 593.1
23.1 345.4 127812947.12834 1282881.1

Then:
A = zeros(100000, 101); % This is ~ 800MB, are you sure you have enough memory?

fid = fopen('data.txt', 'r');
row = 1;

% Skip first line
tline = fgetl(fid);

while 1
  tline = fgetl(fid);
  if ~ischar(tline), break, end
  A(row, :) = str2double(regexp(tline, ' ', 'split'));
end
fclose(fid);

If you have binary data, use fread instead.

Good luck.


"Sven B?" <sven.boehm@contiautomotive.com> wrote in message <giajg4$f3t$1@fred.mathworks.com>...
> Hi,
> from my PSPICE-simulation I get a text-file with the simulated data.
> The text file is approx. 500MB.
> The data in the file consists of 101 columns and approx. 100000 rows.
> For Matlab it is not possible to import this file via the Import Wizard.
> The first row is normal text, the others should be numbers.
> The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
> Each column is the result of a Monte Carlo analysis.
>
> The Import wizard starts up an then hangs on the point "creating preview".
>
> Does anyone can help me?
>
> Regards,
>
> Sven

Subject: Problems with importing large text files

From: Andres

Date: 17 Dec, 2008 11:06:02

Message: 3 of 4

"Sven B?" <sven.boehm@contiautomotive.com> wrote in message <giajg4$f3t$1@fred.mathworks.com>...
> Hi,
> from my PSPICE-simulation I get a text-file with the simulated data.
> The text file is approx. 500MB.
> The data in the file consists of 101 columns and approx. 100000 rows.
> For Matlab it is not possible to import this file via the Import Wizard.
> The first row is normal text, the others should be numbers.
> The first row has no special delimiter. Sometimes there are 2 sometimes 4 blanks as delimiter.
> Each column is the result of a Monte Carlo analysis.
>
> The Import wizard starts up an then hangs on the point "creating preview".

Seems like the import wizard is overchallenged... but i guess you can go without it. You obviously know everthing about the the file layout, so a textscan (+ fgetl) should do, like

nCol = 101;
fid = fopen('myfile.txt');
firstRowString = fgetl(fid);
data = textscan(fid, repmat('%f',1,nCol),'CollectOutput', true);
fclose(fid)

Maybe you have to adjust for the delimiters in the data section or other details, see the textscan doc. I *assume* you'll get no memory issues in this way.

Subject: Problems with importing large text files

From: Sven B?

Date: 17 Dec, 2008 11:59:04

Message: 4 of 4

Thanks to both of you.
The one from Andres was the thing that saved my day....

Sven

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us