MATLAB Central Newsreader - breaking up a large matrix before loading into matlab

breaking up a large matrix before loading into matlab

gary12345 — Thu, 02 Jul 2009 21:04:38 -0400

Does anybody know how I can load parts of a matrix into matlab instead of all at once? because matlab won't allow me to load it all at once because the matrix is too big. Thanks.

Re: breaking up a large matrix before loading into matlab

dpb — Thu, 02 Jul 2009 23:41:52 -0400

gary12345 wrote:
> Does anybody know how I can load parts of a matrix into matlab
> instead of all at once? because matlab won't allow me to load it all
> at once because the matrix is too big. Thanks.

If it's a binary or text file see the doc for the appropriate input
function--they have optional arguments to include number of elements to
read.

If some other ML system did a SAVE to a .MAT file, there I'm not sure
outside a mex file or other hack to get at the data around the structure
of the file that there is...

--

Re: breaking up a large matrix before loading into matlab

Sprinceana — Fri, 03 Jul 2009 06:28:01 -0400

For example if your big matrix (stored in variable) is 720x960 and you want to load only 10*10:

img=imread('poza.jpg');imshow(img);

>>whos img

Name Size Bytes Class

img 720x960 691200 uint8 array

Make this (to store only 10*10 part of that matrix):

small_img=img(1:10,1:10);
imagesc(small_img);

Re: breaking up a large matrix before loading into matlab

us — Fri, 03 Jul 2009 06:44:01 -0400

"Sprinceana " <mihaispr@yahoo.com> wrote in message <h2k8dh$6uc$1@fred.mathworks.com>...
> For example if your big matrix (stored in variable) is 720x960 and you want to load only 10*10:
>
>
> img=imread('poza.jpg');imshow(img);
>
>
> >>whos img
>
> Name Size Bytes Class
>
> img 720x960 691200 uint8 array
>
> Make this (to store only 10*10 part of that matrix):
>
> small_img=img(1:10,1:10);
> imagesc(small_img);

SO... you end up having the BIG -AND- the SMALL matrix in your workspace...
how does this help the OP...

us

Re: breaking up a large matrix before loading into matlab

us — Fri, 03 Jul 2009 06:45:03 -0400

gary12345 <garycyhou@yahoo.ca> wrote in message <18789979.66027.1246568708618.JavaMail.jakarta@nitrogen.mathforum.org>...
> Does anybody know how I can load parts of a matrix into matlab instead of all at once? because matlab won't allow me to load it all at once because the matrix is too big. Thanks.

a hint:

help memmapfile;

us

Re: breaking up a large matrix before loading into matlab

Bruno Luong — Fri, 03 Jul 2009 06:48:01 -0400

dpb <none@non.net> wrote in message <h2jh0c$edq$2@news.eternal-september.org>...
> gary12345 wrote:
> > Does anybody know how I can load parts of a matrix into matlab
> > instead of all at once? because matlab won't allow me to load it all
> > at once because the matrix is too big. Thanks.
>
> If it's a binary or text file see the doc for the appropriate input
> function--they have optional arguments to include number of elements to
> read.
>
> If some other ML system did a SAVE to a .MAT file, there I'm not sure
> outside a mex file or other hack to get at the data around the structure
> of the file that there is...

I have not seen he internal format of MAT file documented anywhere. Not sure how to hack it.

Best recommendation is to find a computer with Matlab and large RAM to break down the file.

Few possible attempts: Try first to load the file after CLEAR ALL command. It is possible that the SAVE command is carried out on the same computer with large data-sharing variables. LOAD - since unshared them - thus will be unable to read it on the very same computer. In this case load individual variables separately - interleaved with clear all - might work. If it doesn't pass, then you are screwed.

Bruno

Re: breaking up a large matrix before loading into matlab

us — Fri, 03 Jul 2009 06:55:04 -0400

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <h2k9j1$l96$1@fred.mathworks.com>...
> dpb <none@non.net> wrote in message <h2jh0c$edq$2@news.eternal-september.org>...
> > gary12345 wrote:
> > > Does anybody know how I can load parts of a matrix into matlab
> > > instead of all at once? because matlab won't allow me to load it all
> > > at once because the matrix is too big. Thanks.
> >
> > If it's a binary or text file see the doc for the appropriate input
> > function--they have optional arguments to include number of elements to
> > read.
> >
> > If some other ML system did a SAVE to a .MAT file, there I'm not sure
> > outside a mex file or other hack to get at the data around the structure
> > of the file that there is...
>
> I have not seen he internal format of MAT file documented anywhere. Not sure how to hack it.
>
> Best recommendation is to find a computer with Matlab and large RAM to break down the file.
>
> Few possible attempts: Try first to load the file after CLEAR ALL command. It is possible that the SAVE command is carried out on the same computer with large data-sharing variables. LOAD - since unshared them - thus will be unable to read it on the very same computer. In this case load individual variables separately - interleaved with clear all - might work. If it doesn't pass, then you are screwed.
>
> Bruno

hm... i didn't think the OP was talking about a MAT-file...
anyhow, here's the doc describing the anatomy of a MAT...

http://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf

us

Re: breaking up a large matrix before loading into matlab

Bruno Luong — Fri, 03 Jul 2009 07:04:02 -0400

"us " <us@neurol.unizh.ch> wrote in message <h2ka08$hoq$1@fred.mathworks.com>...

> here's the doc describing the anatomy of a MAT...
>
> http://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf

Ah great, thank you usrs!!

Bruno

Re: breaking up a large matrix before loading into matlab

Rune Allnor — Fri, 03 Jul 2009 09:31:19 -0400

On 2 Jul, 23:04, gary12345 <garycy...@yahoo.ca> wrote:
> Does anybody know how I can load parts of a matrix into matlab instead of all at once? because matlab won't allow me to load it all at once because the matrix is too big. Thanks.

If you have a wintel multi-core PC, then try this:

- Push <ctrl>-<alt>-<del> to bring up the task manager
- Right-click on the matlab process in the processes view
- Select 'set affinity...' in the pop-up menu
- De-select CPU0
- Press 'OK'

This will cause the matlab process to be run on one
of the other cores available. With a little luck,
the OS and most of the other processes run on CPU0
core, meaning lots of the memory associated with this
core is not available to matlab.

If this doesn't work, try the same again but deselect
some other core.

Rune