Thread Subject: calculating distance using PDIST

I am using PDIST function of Statistics Toolbox to find
Euclidean Distance.
but the memory requirement goes beyond the RAM capacity when
I use matrix size (50000,20).

I am trying following :

a=randn(50000,20);
b=squareform(pdist(a));

How the memory requirement can be minimized?

"amit " <amit_tilwankar@yahoo.com> wrote in message
news:g11555$ihe$1@fred.mathworks.com...
>I am using PDIST function of Statistics Toolbox to find
> Euclidean Distance.
> but the memory requirement goes beyond the RAM capacity when
> I use matrix size (50000,20).
>
> I am trying following :
>
> a=randn(50000,20);
> b=squareform(pdist(a));
>
> How the memory requirement can be minimized?

When you call PDIST on that matrix, the output (which will become the input
to SQUAREFORM) is a 1,249,975,000 element vector, which would require
approximately 9.3 GB of memory. I'm guessing your computer doesn't have
that much memory. Then when you call SQUAREFORM, that creates a
50000-by-50000 matrix as output, which requires _another_ 9.3 GB of memory.

You will need to break your problem into smaller pieces and process each
piece separately. Alternately, if you post the problem you're trying to
solve to the newsgroup (and please post it, don't email it to me directly,
I'll see it when you post it), there may be a way to solve it that doesn't
require creating such a large matrix.

--
Steve Lord
slord@mathworks.com

amit wrote:
> I am using PDIST function of Statistics Toolbox to find
> Euclidean Distance.
> but the memory requirement goes beyond the RAM capacity when
> I use matrix size (50000,20).

Are you sure you have your data in the correct orientation, i.e., 50000
observations, each in 20 dimensions?

As others have said, you are unlikely to be able to fit a 50000x49999/2 distance
matrix upper triangle into memory, and even if you could, what would you do with
such a thing that wouldn't take even _more_ memory?

Thanks for suggestions!

I just want to find out the distance between all the vectors
 i.e. 50000x20 using PDIST, how the operation can be divided
into pieces?

"Steven Lord" <slord@mathworks.com> wrote in message
<g11763$t0v$1@fred.mathworks.com>...
>
> "amit " <amit_tilwankar@yahoo.com> wrote in message
> news:g11555$ihe$1@fred.mathworks.com...
> >I am using PDIST function of Statistics Toolbox to find
> > Euclidean Distance.
> > but the memory requirement goes beyond the RAM capacity when
> > I use matrix size (50000,20).
> >
> > I am trying following :
> >
> > a=randn(50000,20);
> > b=squareform(pdist(a));
> >
> > How the memory requirement can be minimized?
>
> When you call PDIST on that matrix, the output (which will
become the input
> to SQUAREFORM) is a 1,249,975,000 element vector, which
would require
> approximately 9.3 GB of memory. I'm guessing your
computer doesn't have
> that much memory. Then when you call SQUAREFORM, that
creates a
> 50000-by-50000 matrix as output, which requires _another_
9.3 GB of memory.
>
> You will need to break your problem into smaller pieces
and process each
> piece separately. Alternately, if you post the problem
you're trying to
> solve to the newsgroup (and please post it, don't email it
to me directly,
> I'll see it when you post it), there may be a way to solve
it that doesn't
> require creating such a large matrix.
>
> --
> Steve Lord
> slord@mathworks.com
>
>