Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Matlab slows down after apeending to a file

Subject: Matlab slows down after apeending to a file

From: Alex

Date: 8 May, 2012 11:35:06

Message: 1 of 10

Dear all,
I am using matlab R2010b, 32bit version to 1) query a device and 2)store those results to hard disk.

That device, actually is a measurement devices, every one second "feeds" matlab with 256kbytes of data (32.000 number of double precision).

I am appending those measurements to a file and for reducing the extra burden I write those numbers fwrite(fileID, DataToWriteToFile, 'float32') as float32.

After few days the size of the file increases in size (i.e 20Gb after 3 days of non stop getting n storing measurements into a file) and it looks like that the matlab is slowing down. By using tic, toc in my code I have found that this come from the writing part (fwrite).

 I was wondering then if this might be the case as the operating system is 32bit and thus addressing more of a few gigabytes has to start doing "weird" things (like keeping tracks of pointers around for accessing the extra space).

Do you know if there are already mentioned such limitations in literature concerning file limitations in matlab?

I would like to thank you in advance for your reply


B.R
Alex

Subject: Matlab slows down after apeending to a file

From: Bruno Luong

Date: 8 May, 2012 12:01:07

Message: 2 of 10

What OS are you running on?
Which file system is your HD (NTFS, fat32?)
Do you open the file everytime you write?
Do you close it afterward?
How much free space available on the HD?
Do your program happens to fill the RAM as well overtime?
Can you post more details about the way you handle the writing than just the posting fwrite() command?

Bruno

Subject: Matlab slows down after apeending to a file

From: Alex

Date: 8 May, 2012 12:30:08

Message: 3 of 10

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <job1u3$7dj$1@newscl01ah.mathworks.com>...
> What OS are you running on?
> Which file system is your HD (NTFS, fat32?)
> Do you open the file everytime you write?
> Do you close it afterward?
> How much free space available on the HD?
> Do your program happens to fill the RAM as well overtime?
> Can you post more details about the way you handle the writing than just the posting fwrite() command?
>
> Bruno

Sure,

windows xp 32bit, windows 7 32bit. Same problem in same systesm. Filesystem is NTFS
No I open the file once

the code is like that


fopen(append mode)
for 100.000 iterations
       fwrite
end
fclose

I have 100Gb free space and after one week the file size will be around 50Gb

Yes the program does. I am running a 4gb with 1.76Gb constantly free.

My guess is that this has to do with the increase in the file size. It might be much larger of what the operating system can support directly so it has to do "extra" things (like keeping track of pointers)

Regards
Alex

Subject: Matlab slows down after apeending to a file

From: Bruno Luong

Date: 8 May, 2012 13:08:06

Message: 4 of 10

"Alex " <alaios@yahoo.com> wrote in message <job3kg$e7b$1@newscl01ah.mathworks.com>...

> My guess is that this has to do with the increase in the file size. It might be much larger of what the operating system can support directly

That does not make sense, because if it's the case you'll get an error not a slow-down.

>so it has to do "extra" things (like keeping track of pointers)

No the pointer is conserved if the file is not close and write. What takes time is perhaps the allocation table must be updated. This allocation table is growing when the file size. This is part of OS management, not MATLAB.

What you could try is saving your results/data in many smaller files (relatively) rather than in a big one.

Bruno

Subject: Matlab slows down after apeending to a file

From: Alex

Date: 8 May, 2012 15:22:06

Message: 5 of 10

I would like to thank you for your answer. Could you please suggest me what file sizes should I start trying with? 1Gb, 10Gb, 20Gb?

Alex

Subject: Matlab slows down after apeending to a file

From: Matt J

Date: 8 May, 2012 15:27:07

Message: 6 of 10

"Alex " <alaios@yahoo.com> wrote in message <job3kg$e7b$1@newscl01ah.mathworks.com>...
>
> fopen(append mode)
> for 100.000 iterations
> fwrite
> end
> fclose
>
> I have 100Gb free space and after one week the file size will be around 50Gb
==============

You should probably pre-allocate the disk space consumed by the file. Create a 50Gb file and then overwrite relevant portions of it with your for-loop.

Subject: Matlab slows down after apeending to a file

From: Matt J

Date: 8 May, 2012 15:33:06

Message: 7 of 10

"Matt J" wrote in message <jobe0b$4b5$1@newscl01ah.mathworks.com>...
>
> You should probably pre-allocate the disk space consumed by the file. Create a 50Gb file and then overwrite relevant portions of it with your for-loop.
================

Incidentally, I've never heard of anyone creating a single file of a size like 50GB. You're doing some pretty off-the-wall stuff, by the standards of my sheltered world anyway...

Subject: Matlab slows down after apeending to a file

From: Alex

Date: 8 May, 2012 15:36:07

Message: 8 of 10

"Matt J" wrote in message <jobe0b$4b5$1@newscl01ah.mathworks.com>...
> "Alex " <alaios@yahoo.com> wrote in message <job3kg$e7b$1@newscl01ah.mathworks.com>...
> >
> > fopen(append mode)
> > for 100.000 iterations
> > fwrite
> > end
> > fclose
> >
> > I have 100Gb free space and after one week the file size will be around 50Gb
> ==============
>
> You should probably pre-allocate the disk space consumed by the file. Create a 50Gb file and then overwrite relevant portions of it with your for-loop.


Thats also a nice suggestion.
a. How I can preallocate that in matlab?
b. How I can know in advance how much large this file should be? I know in advance that I will loop 800.000 times and each loop produce 32.000 singles. Is that only written in a matlab file or I have also to take care for some extra space that matlab needs to save in that file?

Regards
Alex

Subject: Matlab slows down after apeending to a file

From: Matt J

Date: 8 May, 2012 18:15:07

Message: 9 of 10

"Alex " <alaios@yahoo.com> wrote in message <jobeh7$6ud$1@newscl01ah.mathworks.com>...
> "Matt J" wrote in message <jobe0b$4b5$1@newscl01ah.mathworks.com>...
> > "Alex " <alaios@yahoo.com> wrote in message <job3kg$e7b$1@newscl01ah.mathworks.com>...
> > >
> > > fopen(append mode)
> > > for 100.000 iterations
> > > fwrite
> > > end
> > > fclose
> > >
> > > I have 100Gb free space and after one week the file size will be around 50Gb
> > ==============
> >
> > You should probably pre-allocate the disk space consumed by the file. Create a 50Gb file and then overwrite relevant portions of it with your for-loop.
>
>
> Thats also a nice suggestion.

> b. How I can know in advance how much large this file should be? I know in advance that I will loop 800.000 times and each loop produce 32.000 singles. Is that only written in a matlab file or I have also to take care for some extra space that matlab needs to save in that file?
============

Your story seems to be changing. Earlier, you said you were looping 100000 times and now you're looping 800000. This leads to a memory consumption of

800000*32000*4/2^30 = 95.3674 GB

which is almost all of the free space on your 100GB hard drive. I'm even less certain that whatever you're doing is viable.

In any case, if you're writing to a binary file (not a .mat) then no additional space in the file is required.

> a. How I can preallocate that in matlab?
============

Well, on 32-bit, the maximum MATLAB array size you can allocate is about 0.5GB. That means that you'll need about 200 iterations of fwrite to create a 95 GB template file (e.g., full of zeros). This will probably take some time, but if you can buy a bigger hard drive, you only have to create the template file once and just make a copies of it.

Maybe you can create the template file on a 64 bit machine where the maximum MATLAB array size is larger and then just copy it to your 32 bit hard drive. With a larger array size you could loop in larger chunks.

Subject: Matlab slows down after apeending to a file

From: Bruno Luong

Date: 8 May, 2012 18:47:07

Message: 10 of 10

"Alex " <alaios@yahoo.com> wrote in message <jobdmu$33v$1@newscl01ah.mathworks.com>...
> I would like to thank you for your answer. Could you please suggest me what file sizes should I start trying with? 1Gb, 10Gb, 20Gb?

0.1 Gb.

Bruno

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us