Got Questions? Get Answers.
Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Large matrix - .dat to .mat

Subject: Large matrix - .dat to .mat

From: 19khz

Date: 3 Dec, 2004 02:35:24

Message: 1 of 6

I have done a search of this newsgroup and came across the following
which details a way of handling a large matrix and writing from ascii
to binary.

http://groups-beta.google.com/group/comp.soft-sys.matlab/browse_thread/thread/3ad73ff54e0164cc?hl=en&lr=&ie=UTF-8&oe=UTF-8

Unfortunately, I cannot get it to work successfully. The code executes
fine, put when I want to load the .mat file into matlab, it claims it
isn't able to read it.

Here is a sample of the source file:

0.00000200 0.001243
0.00000400 -0.000775
0.00000600 0.000153
0.00000800 0.000001
0.00001000 -0.001096
0.00001200 0.000469
0.00001400 -0.001404
0.00001600 0.001248

And here is the (modified from link above) code I am using:

fid=fopen('input.dat'); %source file
fid2=fopen('output.mat','wb'); %target file
nrows = 40;%49990000;
for i=1:nrows; %slow loop
str=fgetl(fid); %read a row from the text file as string
values = strread(str,'%.10f'); %string->nums
fwrite(fid2,values,'float'); %write a row to the binary file
end
fclose(fid); %close the text file
fclose(fid2); %close the binary file

As you can see in the third line, there are nearly 50 million lines
(doing initial test with 40 lines), so I have to approach it using the
line by line method to avoid running out of memory; unless there is
another easier method?? The source file is 1GB in size.
Does anyone know where I am going wrong?

Kind regards
19kHz

Subject: Large matrix - .dat to .mat

From: Yi Cao

Date: 3 Dec, 2004 05:57:39

Message: 2 of 6

19khz wrote:
>
>
> I have done a search of this newsgroup and came across the
> following
> which details a way of handling a large matrix and writing from
> ascii
> to binary.
>
> <http://groups-beta.google.com/group/comp.soft-sys.matlab/browse_thread/thread/3ad73ff54e0164cc?hl=en&lr=&ie=UTF-8&oe=UTF-8>
>
> Unfortunately, I cannot get it to work successfully. The code
> executes
> fine, put when I want to load the .mat file into matlab, it claims
> it
> isn't able to read it.
>
> Here is a sample of the source file:
>
> 0.00000200 0.001243
> 0.00000400 -0.000775
> 0.00000600 0.000153
> 0.00000800 0.000001
> 0.00001000 -0.001096
> 0.00001200 0.000469
> 0.00001400 -0.001404
> 0.00001600 0.001248
>
> And here is the (modified from link above) code I am using:
>
> fid=fopen('input.dat'); %source file
> fid2=fopen('output.mat','wb'); %target file
> nrows = 40;%49990000;
> for i=1:nrows; %slow loop
> str=fgetl(fid); %read a row from the text file as string
> values = strread(str,'%.10f'); %string->nums
> fwrite(fid2,values,'float'); %write a row to the binary file
> end
> fclose(fid); %close the text file
> fclose(fid2); %close the binary file
>
> As you can see in the third line, there are nearly 50 million lines
> (doing initial test with 40 lines), so I have to approach it using
> the
> line by line method to avoid running out of memory; unless there is
> another easier method?? The source file is 1GB in size.
> Does anyone know where I am going wrong?
>
> Kind regards
> 19kHz
>
>

Have tried MATLAB "Import Data" utility?
From File menu click Import Data.

Subject: Large matrix - .dat to .mat

From: Tristram Scott

Date: 3 Dec, 2004 11:51:46

Message: 3 of 6

19khz wrote:
>
> I have done a search of this newsgroup and came across the following
> which details a way of handling a large matrix and writing from ascii
> to binary.
>
> http://groups-beta.google.com/group/comp.soft-sys.matlab/browse_thread/thread/3ad73ff54e0164cc?hl=en&lr=&ie=UTF-8&oe=UTF-8
>
> Unfortunately, I cannot get it to work successfully. The code executes
> fine, put when I want to load the .mat file into matlab, it claims it
> isn't able to read it.
>
> Here is a sample of the source file:
>
> 0.00000200 0.001243
> 0.00000400 -0.000775
> 0.00000600 0.000153
> 0.00000800 0.000001
> 0.00001000 -0.001096
> 0.00001200 0.000469
> 0.00001400 -0.001404
> 0.00001600 0.001248
>
> And here is the (modified from link above) code I am using:
>
> fid=fopen('input.dat'); %source file
> fid2=fopen('output.mat','wb'); %target file
> nrows = 40;%49990000;
> for i=1:nrows; %slow loop
> str=fgetl(fid); %read a row from the text file as string
> values = strread(str,'%.10f'); %string->nums
> fwrite(fid2,values,'float'); %write a row to the binary file
> end
> fclose(fid); %close the text file
> fclose(fid2); %close the binary file
>
> As you can see in the third line, there are nearly 50 million lines
> (doing initial test with 40 lines), so I have to approach it using the
> line by line method to avoid running out of memory; unless there is
> another easier method?? The source file is 1GB in size.
> Does anyone know where I am going wrong?
>
> Kind regards
> 19kHz


MATLAB will load ASCII or native MATLAB data files using the load
command. To load a binary file, use fread instead:


>> fid = fopen('output.mat','r');
>> x = fread(fid,inf,'float');
>> fclose(fid);

So the extension of .mat is not really appropriate, unless you are also
going to write the header informationm MATLAB expects.

The load command may well be able to open your .dat file for you:

>> x = load('input.dat');

Since you intend to load it into MATLAB, I assume you have enough RAM to
cope. You will need 800 MB for the array (50,000,000 x 2 x 8 bytes per
element). That being the case, if load won't grab it in for you, you
can read it a line at a time (or maybe a bunch of lines at a time might
be faster) and stuff these into the array:

nrows = 40;%49990000;
x = zeros(nrows,2);
fid=fopen('input.dat'); %source file
for i=1:nrows; %slow loop
str=fgetl(fid); %read a row from the text file as string
values = strread(str,'%.10f'); %string->nums
x(i,:) = values';
end
fclose(fid); %close the text file

--
Dr Tristram J. Scott
Energy Consultant

Subject: Large matrix - .dat to .mat

From: 19khz

Date: 6 Dec, 2004 01:28:17

Message: 4 of 6

Thank you for your time Tristram.

The reason I want it in .mat format (matlab's native binary) is because
I need to use it in Simulink. I don't know whether there is any other
way of reading in binary data into simulink other than matlab's own. I
was just using the load command to test the binary file to see if
Matlab could read it.

I have tried loading the dat file into matlab using:

>>load input.dat

but it is unfortunately too big. If I could load it, there would be no
problem in saving as Matlabs native binary. Hence my need to do it
manually.

So, with that information in mind, it appears I may have to write the
header information myself with this:
http://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf

If you have any other suggestions, I would be grateful.

And finally, I only have 512MB RAM, and probably only half that
available!

TIA
Kind rgds
19k.

Subject: Large matrix - .dat to .mat

From: 19khz

Date: 6 Dec, 2004 01:54:46

Message: 5 of 6

> Have tried MATLAB "Import Data" utility?

I have, but it unfortunately does not overcome the problem of running
out of memory.

Thank you for your time.

rgds
19k.

Subject: Large matrix - .dat to .mat

From: Steve Amphlett

Date: 6 Dec, 2004 06:25:37

Message: 6 of 6

19kHz wrote:
>
>
>> Have tried MATLAB "Import Data" utility?
>
> I have, but it unfortunately does not overcome the problem of
> running
> out of memory.

I would say that your options are limited. In earlier versions of
ML, general file parsing was virtually impossible. So we were forced
to write dedicated C-MEX files. While they were time-consuming to
write and a PITA to support, they provided functionality that wasn't
there in base ML. And they were quick (a lot quicker than
stream-editing the data and then using <load> on a resulting
flat file).

It sounds like you have insufficient memory for the "load it all in
and parse it in memory" approach. And two much data to parse in a ML
loop. That leaves two approaches:

1) A C-MEX function (not really that difficult).

2) A .dat->.mat translator. ML publish the .mat file format and
provide tools for writing them.

Personally I'd go for option 1 (based more on familiarity than
anything objective).

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us