Thread Subject: Is there a way to create > 4 GB .mat file

Hello,


I want to append an existing .mat files with lots of variables. Each variable is ~100 MB and the total disk space after appending all variables could grow to more than 50 GB!

I have tried doing this using 'save -append' in Matlab 2009b but it somehow restricts the file to 4 GB.

Is there a work-around ?

Regards,
Sid

I have no problems creating 8 GB MAT files. 2 possible reasons why you cannot:

1) You are using 32-bit Matlab
2) You are using a filesystem which cannot create files >4GB (e.g. FAT32 on Windows or (old) ext2 without feature LARGE_FILES).

Sebastiaan


"Verma " <siddharth.verma@db.com> wrote in message <hdilnq$rc4$1@fred.mathworks.com>...
> Hello,
>
>
> I want to append an existing .mat files with lots of variables. Each variable is ~100 MB and the total disk space after appending all variables could grow to more than 50 GB!
>
> I have tried doing this using 'save -append' in Matlab 2009b but it somehow restricts the file to 4 GB.
>
> Is there a work-around ?
>
> Regards,
> Sid

Hello,
Is it possible to install 64 bit Matlab on 32 bit OS?
Ankit

"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl> wrote in message <hdj5ke$8ip$1@fred.mathworks.com>...
> I have no problems creating 8 GB MAT files. 2 possible reasons why you cannot:
>
> 1) You are using 32-bit Matlab
> 2) You are using a filesystem which cannot create files >4GB (e.g. FAT32 on Windows or (old) ext2 without feature LARGE_FILES).
>
> Sebastiaan
>
>
> "Verma " <siddharth.verma@db.com> wrote in message <hdilnq$rc4$1@fred.mathworks.com>...
> > Hello,
> >
> >
> > I want to append an existing .mat files with lots of variables. Each variable is ~100 MB and the total disk space after appending all variables could grow to more than 50 GB!
> >
> > I have tried doing this using 'save -append' in Matlab 2009b but it somehow restricts the file to 4 GB.
> >
> > Is there a work-around ?
> >
> > Regards,
> > Sid

"Ankit " <ankit-a.jain@db.com> wrote in message <hdja0g$llv$1@fred.mathworks.com>...
> Hello,
> Is it possible to install 64 bit Matlab on 32 bit OS?
> Ankit
>
No, because there is no way your system is able to use/understand 64bit pointers. Not even if your CPU supports 64 bit, when a 32 bit OS is installed.

You can do something like this:

NoChunks = 1;
try
  save(sprintf('data_chunk_%d', NoChunks), '-append', 'a','b','c');
catch
  NoChunks = NoChunks + 1;
  save(sprintf('data_chunk_%d', NoChunks), 'a','b','c');
end

So, if it fails saving, it creates a new file. If you can figure out the error idenfier when saving, you can make it more robust (i.e. it only tries saving it to a new file if the file size limit is exceeded, but not if the disk is full).

Sebastiaan

"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl> wrote in message
news:hdjjvb$7iv$1@fred.mathworks.com...
> "Ankit " <ankit-a.jain@db.com> wrote in message
> <hdja0g$llv$1@fred.mathworks.com>...
>> Hello,
>> Is it possible to install 64 bit Matlab on 32 bit OS?
>> Ankit
>>
> No, because there is no way your system is able to use/understand 64bit
> pointers. Not even if your CPU supports 64 bit, when a 32 bit OS is
> installed.
>
> You can do something like this:
>
> NoChunks = 1;
> try
> save(sprintf('data_chunk_%d', NoChunks), '-append', 'a','b','c');
> catch
> NoChunks = NoChunks + 1;
> save(sprintf('data_chunk_%d', NoChunks), 'a','b','c');
> end
>
> So, if it fails saving, it creates a new file. If you can figure out the
> error idenfier when saving, you can make it more robust (i.e. it only
> tries saving it to a new file if the file size limit is exceeded, but not
> if the disk is full).
>
> Sebastiaan

In addition to Sebastiaan's comments, you should consider writing the
variables to individual MAT-files. That way you don't put all your eggs in
one basket (if something happens to corrupt the file while it's saving a
variable, you've lost just the one variable not all the variables in the
file) and it's easier to back up lots of relatively small MAT-files (via CD,
DVD, flash drive, etc.) than it is to back up one monolithic 50 GB MAT-file.

--
Steve Lord
slord@mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ