Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
MAT file takes forever to save with big object array

Subject: MAT file takes forever to save with big object array

From: Alexey

Date: 21 Dec, 2010 09:34:33

Message: 1 of 7

Hi,

I have a 65k-element array of objects of a particular class.

>> whos results
  Name Size Bytes Class Attributes

  results 1x65000 3025921848 StrategyResultDef

When I try to save it to a MAT file I get the following error:

>save backtesting/files/technical-obj
Warning: Variable 'results' cannot be saved to a MAT-file whose version is older
than 7.3.
To save this variable, use the -v7.3 switch.
Skipping...

And the variable 'results' is not saved.

If I do:

>save -v7.3 backtesting/files/technical-obj

I do not get an error but it takes about 4 hours to save the file and its size is 1.5gb.

I want to save the results for later processing but at this rate it's easier for me to re-generate the results every time.

I will have a look at reducing the size of the structure but does anyone have any suggestions as to what is the best way to save the results like this?

Kind Regards,
Alexey

Subject: MAT file takes forever to save with big object array

From: Walter Roberson

Date: 21 Dec, 2010 17:17:38

Message: 2 of 7

On 21/12/10 3:34 AM, Alexey wrote:

>> save backtesting/files/technical-obj

> To save this variable, use the -v7.3 switch.

>> save -v7.3 backtesting/files/technical-obj
>
> I do not get an error but it takes about 4 hours to save the file and
> its size is 1.5gb.

> I will have a look at reducing the size of the structure but does anyone
> have any suggestions as to what is the best way to save the results like
> this?

You didn't have to start a new thread; you could just have replied to
the old thread that indeed you were using objects .

As you are using objects, then you need to discuss with the
administrators of the HPC site how to save the file on a drive that is
local to the CPU you are using rather than being on a networked file system.

There has been no information from Mathworks posted on this forum as to
_why_ the -v7.3 format can be as slow as it is, but there have been
enough reports from users that we know that this is a fact of life for
some kinds of data on some systems, at least up to 2010a. (People
haven't happened to discuss it for 2010b.)

Subject: MAT file takes forever to save with big object array

From: Sean de

Date: 21 Dec, 2010 17:48:05

Message: 3 of 7

"Alexey" wrote in message <iepsb9$mrm$1@fred.mathworks.com>...
> Hi,
>
> I have a 65k-element array of objects of a particular class.
>
> >> whos results
> Name Size Bytes Class Attributes
>
> results 1x65000 3025921848 StrategyResultDef
>
> When I try to save it to a MAT file I get the following error:
>
> >save backtesting/files/technical-obj
> Warning: Variable 'results' cannot be saved to a MAT-file whose version is older
> than 7.3.
> To save this variable, use the -v7.3 switch.
> Skipping...
>
> And the variable 'results' is not saved.
>
> If I do:
>
> >save -v7.3 backtesting/files/technical-obj
>
> I do not get an error but it takes about 4 hours to save the file and its size is 1.5gb.
>
> I want to save the results for later processing but at this rate it's easier for me to re-generate the results every time.
>
> I will have a look at reducing the size of the structure but does anyone have any suggestions as to what is the best way to save the results like this?
>
> Kind Regards,
> Alexey

Hmm that does seem like a really long time. I frequently save/load 9GB files (structs with 3-3GB fields) and it usually takes a few minutes (MAC r2009b, 64bit). I have found for simple computations it's quicker to recalculate than to save. I.e: write a script that loads/generates what you need, run it, go get a coffee.

You could also try writing it to a binary file using fwrite and reading it in with fread. This is a pain though since you need to remember: dimensions, classes, bytes to skip if necessary etc.

Subject: MAT file takes forever to save with big object array

From: Alexey

Date: 26 Dec, 2010 23:36:04

Message: 4 of 7

"Sean de " <sean.dewolski@nospamplease.umit.maine.edu> wrote in message <ieqp8l$sdq$1@fred.mathworks.com>...
> "Alexey" wrote in message <iepsb9$mrm$1@fred.mathworks.com>...
> > Hi,
> >
> > I have a 65k-element array of objects of a particular class.
> >
> > >> whos results
> > Name Size Bytes Class Attributes
> >
> > results 1x65000 3025921848 StrategyResultDef
> >
> > When I try to save it to a MAT file I get the following error:
> >
> > >save backtesting/files/technical-obj
> > Warning: Variable 'results' cannot be saved to a MAT-file whose version is older
> > than 7.3.
> > To save this variable, use the -v7.3 switch.
> > Skipping...
> >
> > And the variable 'results' is not saved.
> >
> > If I do:
> >
> > >save -v7.3 backtesting/files/technical-obj
> >
> > I do not get an error but it takes about 4 hours to save the file and its size is 1.5gb.
> >
> > I want to save the results for later processing but at this rate it's easier for me to re-generate the results every time.
> >
> > I will have a look at reducing the size of the structure but does anyone have any suggestions as to what is the best way to save the results like this?
> >
> > Kind Regards,
> > Alexey
>
> Hmm that does seem like a really long time. I frequently save/load 9GB files (structs with 3-3GB fields) and it usually takes a few minutes (MAC r2009b, 64bit). I have found for simple computations it's quicker to recalculate than to save. I.e: write a script that loads/generates what you need, run it, go get a coffee.
>
> You could also try writing it to a binary file using fwrite and reading it in with fread. This is a pain though since you need to remember: dimensions, classes, bytes to skip if necessary etc.


Thanks for suggestions!
I have tried to use R2010b and it has the same problem. Once the vector of objects reaches 64k elements it is impossible to save it. Just takes forever and then fails with "cannot close file" or something like this.

I'll probably consider saving to database as I wanted to do searches through the results.

Subject: MAT file takes forever to save with big object array

From: Jeremy

Date: 13 Apr, 2011 17:19:04

Message: 5 of 7

Alexey - I am having the exact same problem as you. As a partial solution, I tried using serialize/deserialize (from the FEX) before save. With a ~2.5GB cell array of structures, the command save('blah.mat',serialize(str),'-v7.3') took about a minute, as opposed to a couple of hours with save and v7.3 alone. Unfortunately, for me this only worked with variables of around 2.5GB in size. Trying the same approach with a 8.5GB cell array of structures never quite finished serializing/saving before I got frustrated and Ctrl+C'd it.

Subject: MAT file takes forever to save with big object array

From: J RP

Date: 14 Oct, 2012 23:18:07

Message: 6 of 7

Same sort of problem here.

I can save a 4-D double array (initialized by zeros(k,l,m,n)) that ends up being 2612736000 bytes in about 1 minute. Then I try to save a structure with two fields--one field is a vector and another contains one or more .fit class objects--and it's only 184268260 bytes but takes literally hours and hours to save just a fraction of this struct variable and I end up killing the process because it takes longer to save the data than just generate it, even though it takes 3 hrs to generate it!

Why would this be the case? It makes little sense.

Subject: MAT file takes forever to save with big object array

From: J

Date: 6 Dec, 2012 19:34:08

Message: 7 of 7

Walter Roberson <roberson@hushmail.com> wrote in message
>
>
> As you are using objects, then you need to discuss with the
> administrators of the HPC site how to save the file on a drive that is
> local to the CPU you are using rather than being on a networked file system.
>
> There has been no information from Mathworks posted on this forum as to
> _why_ the -v7.3 format can be as slow as it is, but there have been
> enough reports from users that we know that this is a fact of life for
> some kinds of data on some systems, at least up to 2010a. (People
> haven't happened to discuss it for 2010b.)
>

I can verify that this isn't just a problem for 2010a and before, and isn't just a problem for saving to networked drives. I have a 60 MB matrix of .fit objects that takes hours and hours and hours to save to the local disk in 2010b. Luckily, after it completes saving after a ridiculously long time (this happens whether I call with -v7.3 or not), it only takes 30 sec or so to load in the future, so I don't have to re-compute the data every time.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us