Thread Subject: Data is larger in memory than on harddrive

Hi all!

I have a very strange behaviour in the last time.
I'm loading several big matrices(385x2811x7x9) from files
into an array.

The files are all together 350 MB large but the array takes
530 MB!
The weirdest thing is that I'm getting an error message "Out
of memory" even though it only uses ~1.2 Gig of the 2 GB
available!
Furthermore I used to run this code on a 1 GB machine and I
only changed a tiny bit of code which actually generates the
same data...

thanks for your thoughts!
Volker Klink

Sorry guys!
I'm stupid! I've found my error myself...at some point I
created some matrices with the size [2811 2811] instead of
[384 2811]...

Volker

Well..apparently one problem remains:
Even though I have 2 GB RAM it stops at ~1.2 GB telling me
that I don't have enough memory!

What could be wrong?

"Volker K" <klinkv.NOSPAM@yahoo.de> wrote in message
news:fte37t$6np$1@fred.mathworks.com...
> Well..apparently one problem remains:
> Even though I have 2 GB RAM it stops at ~1.2 GB telling me
> that I don't have enough memory!
>
> What could be wrong?

Read this:

http://www.mathworks.com/support/tech-notes/1100/1106.html

How much total memory you have is not the issue. The size of the largest
contiguous block of memory you have available is the issue.

--
Steve Lord
slord@mathworks.com

> How much total memory you have is not the issue. The size
of the largest
> contiguous block of memory you have available is the issue.

Hi Steven!
Thanks for the hint...I've played around with the "pack"
function: even though I've packed my memory and cleared it
completely it won't create a memory block of the same size
as I had when starting up MatLab.

Biggest block on startup: ~900MB
biggest block after loading a lot of data, packing AND
"clear all": ~250 MB


Is there anything you can do about it?
Thansk
Volker Klink

Why areyou unnecessarily suffocating the workspace with huge data? Which processor will process 500MB data in 1 ftech cycle? Are you simply interested in seeing error messages?

> Why areyou unnecessarily suffocating the workspace with
huge data?

I'm not unnecessarily filling up the workspace. I'm working
with large satellite images (384x2811) with 9 different
spectral channels. One image shows a certain stripe of the
earth.For example if I would like to load all 7 scans (which
show the whole earth disc) with all spectral channels I'll
get 7 x 9 x 384 x 2811 = 68003712 elements.
In double that would take 520 MB of memory...

Yumnam Kirani Singh <kirani.singh@gmail.com> wrote in
message
<8573631.1207654364020.JavaMail.jakarta@nitrogen.mathforum.org>...
> Why areyou unnecessarily suffocating the workspace with
huge data? Which processor will process 500MB data in 1
ftech cycle? Are you simply interested in seeing error messages?

That's a rather trite comment to make. If you have nothing
useful to say, please don't say it. Half a gigabyte of data
in the MATLAB workspace is not that big, and anyway, why
should you be surprised that MATLAB can be used to solve
large problems? It can solve larger problems than this - I
do it all the time (with a tad bit more memory that the
Original Poster has, though).

The reason the file is smaller than the matrix in memory is
because of file compression. *.mat files are compressed
when they are saved to disk, and uncompressed when read back
in. The difference can be dramatic, particularly if your
values are not, say, from "rand" but have patterns in them,
repeated values, are of lower precision than double, etc.

You might try working in single.

Windows doesn't do a very good job of memory management,
unfortunately. The Mac is only marginally better; Linux and
Solaris are far better. Failing that ... you could try to
buy more memory.

Sorry if my message below wasn't clear. I changed who I was
address in the middle of my note. In the first part I'm
replying to the spurious comment from Yumnam Singh:

"Tim Davis" <davis@cise.ufl.edu> wrote in message
<ftg027$6fh$1@fred.mathworks.com>...
> Yumnam Kirani Singh <kirani.singh@gmail.com> wrote in
> message
>
<8573631.1207654364020.JavaMail.jakarta@nitrogen.mathforum.org>...
> > Why areyou unnecessarily suffocating the workspace with
> huge data? Which processor will process 500MB data in 1
> ftech cycle? Are you simply interested in seeing error
messages?
>
> That's a rather trite comment to make. If you have nothing
> useful to say, please don't say it. Half a gigabyte of data
> in the MATLAB workspace is not that big, and anyway, why
> should you be surprised that MATLAB can be used to solve
> large problems? It can solve larger problems than this - I
> do it all the time (with a tad bit more memory that the
> Original Poster has, though).

At this point, I'm reply to Volker, not the spurious comment:

>
> The reason the file is smaller than the matrix in memory is
> because of file compression. *.mat files are compressed
> when they are saved to disk, and uncompressed when read back
> in. The difference can be dramatic, particularly if your
> values are not, say, from "rand" but have patterns in them,
> repeated values, are of lower precision than double, etc.
>
> You might try working in single.
>
> Windows doesn't do a very good job of memory management,
> unfortunately. The Mac is only marginally better; Linux and
> Solaris are far better. Failing that ... you could try to
> buy more memory.