Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
mat file gets saved but too big to read

Subject: mat file gets saved but too big to read

From: Misha Koshelev

Date: 1 Sep, 2009 01:07:19

Message: 1 of 7

Dear All:

I am having a problem. I seem to be able to save my mat
file but am getting the error
"Can't read file"
when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you

Misha

Subject: mat file gets saved but too big to read

From: ImageAnalyst

Date: 1 Sep, 2009 01:46:47

Message: 2 of 7

On Aug 31, 9:07 pm, "Misha Koshelev" <mk144...@bcm.edu> wrote:
> Dear All:
>
> I am having a problem. I seem to be able to save my mat
> file but am getting the error
> "Can't read file"
> when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you
>
> Misha

----------------------------------------------------
Could you supply more info, such as just how large is it (file size on
disk) when it works and when it fails, and what does it say when you
type memory on the command line? And what does "# of draws" mean?
And show us your code when you call save() and load().

Subject: mat file gets saved but too big to read

From: Misha Koshelev

Date: 1 Sep, 2009 16:49:02

Message: 3 of 7

ImageAnalyst <imageanalyst@mailinator.com> wrote in message <64d35661-93f3-48a3-9fa6-7334ac4f1c7c@d21g2000vbm.googlegroups.com>...
> On Aug 31, 9:07?pm, "Misha Koshelev" <mk144...@bcm.edu> wrote:
> > Dear All:
> >
> > I am having a problem. I seem to be able to save my mat
> > file but am getting the error
> > "Can't read file"
> > when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you
> >
> > Misha
>
> ----------------------------------------------------
> Could you supply more info, such as just how large is it (file size on
> disk) when it works and when it fails, and what does it say when you
> type memory on the command line? And what does "# of draws" mean?
> And show us your code when you call save() and load().

Thank you. I think the save and load will be a bit much to show as my code is very long. However, I have split up the structures and still getting the same problem. I am wondering if it is a setup issue for Windows vs Unix, as on my Unix machine I can load the same file just fine.

One file:
-rwxrwSrwx 1 misha misha 238523924 2009-08-31 22:00 param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat*

Approx 240 MB (should load on either Windows machine I would think - 3 GB of RAM).

Here it is on a Unix machine (draws are for a Gibbs sampler - sorry):
>> load param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
>> who

Your variables are:

fixed_param

>> fixed_param

fixed_param =

3001x3 struct array with fields:
    theta
    tau
    N_k
    vec
    Y
    Z
    Z_transpose

>>

misha@misha-d630:/tmp/a/many-draws2$ free
             total used free shared buffers cached
Mem: 8193788 6337544 1856244 0 7976 3991720
-/+ buffers/cache: 2337848 5855940
Swap: 0 0 0

You will note 8 GB on the Unix machine, but 3.5 are being used by VMWare-d Windows. I will post Windows output shortly.

Thank you
Misha

Subject: mat file gets saved but too big to read

From: Misha Koshelev

Date: 1 Sep, 2009 16:53:02

Message: 4 of 7

ImageAnalyst <imageanalyst@mailinator.com> wrote in message <64d35661-93f3-48a3-9fa6-7334ac4f1c7c@d21g2000vbm.googlegroups.com>...
> On Aug 31, 9:07?pm, "Misha Koshelev" <mk144...@bcm.edu> wrote:
> > Dear All:
> >
> > I am having a problem. I seem to be able to save my mat
> > file but am getting the error
> > "Can't read file"
> > when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you
> >
> > Misha
>
> ----------------------------------------------------
> Could you supply more info, such as just how large is it (file size on
> disk) when it works and when it fails, and what does it say when you
> type memory on the command line? And what does "# of draws" mean?
> And show us your code when you call save() and load().

On the Windows machine:

  To get started, type one of these: helpwin, helpdesk, or demo.
  For product information, visit www.mathworks.com.
 
? cd E:
? cd many-draws2
? ls

.
..
backup-11360
backup-17560
backup-23760
backup-29960
backup-30001
backup-42
backup-5160
hs_err_pid240.log
output
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-12005-15005.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-15006-18006.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-18007-21007.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-21008-24008.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-24009-27009.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-27010-30001.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-3002-6002.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-6003-9003.mat
param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-9004-12004.mat
state-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25.mat


? who

Your variables are:

ans

? load param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
??? Error using ==> load
Out of memory. Type HELP MEMORY for your options.
 
? feature(memstats)
??? Undefined function or variable 'memstats'.
 
? feature('memstats')

    Physical Memory (RAM):
        In Use: 432 MB (1b0ff000)
        Free: 2678 MB (a767d000)
        Total: 3111 MB (c277c000)
    Page File (Swap space):
        In Use: 346 MB (15ae7000)
        Free: 4650 MB (122aa0000)
        Total: 4997 MB (138587000)
    Virtual Memory (Address Space):
        In Use: 2015 MB (7df7d000)
        Free: 32 MB ( 2063000)
        Total: 2047 MB (7ffe0000)
    Largest Contiguous Free Blocks:
         1. [at a540000] 8 MB ( 800000)
         2. [at 7c578000] 0 MB ( b8000)
         3. [at 1ff49000] 0 MB ( b7000)
         4. [at 719ea000] 0 MB ( b6000)
         5. [at 7fdf1000] 0 MB ( af000)
         6. [at 72f57000] 0 MB ( a9000)
         7. [at 76cb8000] 0 MB ( a8000)
         8. [at 7baf8000] 0 MB ( a8000)
         9. [at 762eb000] 0 MB ( a5000)
        10. [at 76bfb000] 0 MB ( 95000)
                                            ======= ==========
                                              14 MB (00e07000)

ans =

     8388608

?

I am stumped as I saved it on same machine in same MATLAB.

Thank you
Misha

Subject: mat file gets saved but too big to read

From: Jan Simon

Date: 1 Sep, 2009 19:13:06

Message: 5 of 7

Dear Misha Koshelev!

How big is the file's contents? Modern Matlab versions writre MAT files with compression, so it if possible, that a 240 MB file has more data than your RAM allows. You can measure the contents size from yout linux system, e.g.
  a = load('File.Mat');
  whos('a')
What's the answer?

Kind regards, Jan

Subject: mat file gets saved but too big to read

From: Michael

Date: 1 Sep, 2009 19:38:03

Message: 6 of 7

"Misha Koshelev" <mk144210@bcm.edu> wrote in message <h7jjhe$ggp$1@fred.mathworks.com>...
> ImageAnalyst <imageanalyst@mailinator.com> wrote in message <64d35661-93f3-48a3-9fa6-7334ac4f1c7c@d21g2000vbm.googlegroups.com>...
> > On Aug 31, 9:07?pm, "Misha Koshelev" <mk144...@bcm.edu> wrote:
> > > Dear All:
> > >
> > > I am having a problem. I seem to be able to save my mat
> > > file but am getting the error
> > > "Can't read file"
> > > when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you
> > >
> > > Misha
> >
> > ----------------------------------------------------
> > Could you supply more info, such as just how large is it (file size on
> > disk) when it works and when it fails, and what does it say when you
> > type memory on the command line? And what does "# of draws" mean?
> > And show us your code when you call save() and load().
>
> On the Windows machine:
>
> To get started, type one of these: helpwin, helpdesk, or demo.
> For product information, visit www.mathworks.com.
>
> ? cd E:
> ? cd many-draws2
> ? ls
>
> .
> ..
> backup-11360
> backup-17560
> backup-23760
> backup-29960
> backup-30001
> backup-42
> backup-5160
> hs_err_pid240.log
> output
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-12005-15005.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-15006-18006.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-18007-21007.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-21008-24008.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-24009-27009.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-27010-30001.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-3002-6002.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-6003-9003.mat
> param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-9004-12004.mat
> state-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25.mat
>
>
> ? who
>
> Your variables are:
>
> ans
>
> ? load param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
> ??? Error using ==> load
> Out of memory. Type HELP MEMORY for your options.
>
> ? feature(memstats)
> ??? Undefined function or variable 'memstats'.
>
> ? feature('memstats')
>
> Physical Memory (RAM):
> In Use: 432 MB (1b0ff000)
> Free: 2678 MB (a767d000)
> Total: 3111 MB (c277c000)
> Page File (Swap space):
> In Use: 346 MB (15ae7000)
> Free: 4650 MB (122aa0000)
> Total: 4997 MB (138587000)
> Virtual Memory (Address Space):
> In Use: 2015 MB (7df7d000)
> Free: 32 MB ( 2063000)
> Total: 2047 MB (7ffe0000)
> Largest Contiguous Free Blocks:
> 1. [at a540000] 8 MB ( 800000)
> 2. [at 7c578000] 0 MB ( b8000)
> 3. [at 1ff49000] 0 MB ( b7000)
> 4. [at 719ea000] 0 MB ( b6000)
> 5. [at 7fdf1000] 0 MB ( af000)
> 6. [at 72f57000] 0 MB ( a9000)
> 7. [at 76cb8000] 0 MB ( a8000)
> 8. [at 7baf8000] 0 MB ( a8000)
> 9. [at 762eb000] 0 MB ( a5000)
> 10. [at 76bfb000] 0 MB ( 95000)
> ======= ==========
> 14 MB (00e07000)
>
> ans =
>
> 8388608
>
> ?
>
> I am stumped as I saved it on same machine in same MATLAB.
>
> Thank you
> Misha

I think one problem may be coming from the fact that your data is stored in a 9000 element array of structs. Is there some way you could break the data down into an n-dimensional array instead? This would get rid of some of the significant overhead that results in using a struct rather than a double array.

Subject: mat file gets saved but too big to read

From: Kevin

Date: 9 Jun, 2011 16:44:04

Message: 7 of 7

"Michael" <michaelmorris1984@gmail.com> wrote in message <h7jt6r$anp$1@fred.mathworks.com>...
> "Misha Koshelev" <mk144210@bcm.edu> wrote in message <h7jjhe$ggp$1@fred.mathworks.com>...
> > ImageAnalyst <imageanalyst@mailinator.com> wrote in message <64d35661-93f3-48a3-9fa6-7334ac4f1c7c@d21g2000vbm.googlegroups.com>...
> > > On Aug 31, 9:07?pm, "Misha Koshelev" <mk144...@bcm.edu> wrote:
> > > > Dear All:
> > > >
> > > > I am having a problem. I seem to be able to save my mat
> > > > file but am getting the error
> > > > "Can't read file"
> > > > when I try to open. It happens only if the file is greater than a certain size (# of draws). Any ideas? Thank you
> > > >
> > > > Misha
> > >
> > > ----------------------------------------------------
> > > Could you supply more info, such as just how large is it (file size on
> > > disk) when it works and when it fails, and what does it say when you
> > > type memory on the command line? And what does "# of draws" mean?
> > > And show us your code when you call save() and load().
> >
> > On the Windows machine:
> >
> > To get started, type one of these: helpwin, helpdesk, or demo.
> > For product information, visit www.mathworks.com.
> >
> > ? cd E:
> > ? cd many-draws2
> > ? ls
> >
> > .
> > ..
> > backup-11360
> > backup-17560
> > backup-23760
> > backup-29960
> > backup-30001
> > backup-42
> > backup-5160
> > hs_err_pid240.log
> > output
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-12005-15005.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-15006-18006.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-18007-21007.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-21008-24008.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-24009-27009.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-27010-30001.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-3002-6002.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-6003-9003.mat
> > param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-9004-12004.mat
> > state-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25.mat
> >
> >
> > ? who
> >
> > Your variables are:
> >
> > ans
> >
> > ? load param-inv-K4-P1-D2-T_idx3-10-RatioData-1-3-4-5-6-7-8-11-20-21-22-23-25-1-3001.mat
> > ??? Error using ==> load
> > Out of memory. Type HELP MEMORY for your options.
> >
> > ? feature(memstats)
> > ??? Undefined function or variable 'memstats'.
> >
> > ? feature('memstats')
> >
> > Physical Memory (RAM):
> > In Use: 432 MB (1b0ff000)
> > Free: 2678 MB (a767d000)
> > Total: 3111 MB (c277c000)
> > Page File (Swap space):
> > In Use: 346 MB (15ae7000)
> > Free: 4650 MB (122aa0000)
> > Total: 4997 MB (138587000)
> > Virtual Memory (Address Space):
> > In Use: 2015 MB (7df7d000)
> > Free: 32 MB ( 2063000)
> > Total: 2047 MB (7ffe0000)
> > Largest Contiguous Free Blocks:
> > 1. [at a540000] 8 MB ( 800000)
> > 2. [at 7c578000] 0 MB ( b8000)
> > 3. [at 1ff49000] 0 MB ( b7000)
> > 4. [at 719ea000] 0 MB ( b6000)
> > 5. [at 7fdf1000] 0 MB ( af000)
> > 6. [at 72f57000] 0 MB ( a9000)
> > 7. [at 76cb8000] 0 MB ( a8000)
> > 8. [at 7baf8000] 0 MB ( a8000)
> > 9. [at 762eb000] 0 MB ( a5000)
> > 10. [at 76bfb000] 0 MB ( 95000)
> > ======= ==========
> > 14 MB (00e07000)
> >
> > ans =
> >
> > 8388608
> >
> > ?
> >
> > I am stumped as I saved it on same machine in same MATLAB.
> >
> > Thank you
> > Misha
>
> I think one problem may be coming from the fact that your data is stored in a 9000 element array of structs. Is there some way you could break the data down into an n-dimensional array instead? This would get rid of some of the significant overhead that results in using a struct rather than a double array.

----------------------------------------------

This is probably a too late to help the original poster, but I ran into the same problem and found a solution. I had done a 'clear' to empty the workspace before loading my data from a file (~500MB) but I was getting the same "Can't read file" errors. Turns out Matlab was still holding on to lots of memory from my old workspace variables. Shutting Matlab down and then doing a load worked for me.

Kevin

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us