MATLAB Central Newsreader - filtering data problem

filtering data problem

Mike — Thu, 23 Apr 2009 00:16:01 -0400

Hi
I usually need to filter data. Sometimes I use logical-indexing, but I found there are many temporary arrays I created. Does that way waste memory? How to avoid this?

Mike

Re: filtering data problem

Jos — Thu, 23 Apr 2009 06:30:07 -0400

"Mike " <sulfateion@gmail.com> wrote in message <gsoc01$eog$1@fred.mathworks.com>...
> Hi
> I usually need to filter data. Sometimes I use logical-indexing, but I found there are many temporary arrays I created. Does that way waste memory? How to avoid this?
>
> Mike

Logical indexing is very powerful but indeed creates a (temporary) array. However, they use 1/8 of the memory occupied by double array of the same size:
A = rand(100,1) ; B = A > 0.5 ; whos

You can clear out, or overwrite, any variables you no longer need:
A = rand(10,1) ;
q = A > 0.5 ;
A = A(q) ; % overwrite
clear q ;

If this is not an option, you can turn to very inefficient loops, like this one, which does not occupy (a lot of) additional memory ...

A = rand(10,1) ;
idx = 1 ;
while (idx <= numel(A)),
  if A(idx) > 0.5,
    idx = idx + 1 ; % goto next element
  else
    A(idx) = [] ; % remove element from list
  end
end

hth
Jos

Re: filtering data problem

Pekka Kumpulainen — Thu, 23 Apr 2009 08:18:01 -0400

"Jos " <#10584@fileexchange.com> wrote in message <gsp1tf$bno$1@fred.mathworks.com>...
> "Mike " <sulfateion@gmail.com> wrote in message <gsoc01$eog$1@fred.mathworks.com>...
> > Hi
> > I usually need to filter data. Sometimes I use logical-indexing, but I found there are many temporary arrays I created. Does that way waste memory? How to avoid this?
> >
> > Mike
>
> Logical indexing is very powerful but indeed creates a (temporary) array. However, they use 1/8 of the memory occupied by double array of the same size:
> A = rand(100,1) ; B = A > 0.5 ; whos
>
> You can clear out, or overwrite, any variables you no longer need:
> A = rand(10,1) ;
> q = A > 0.5 ;
> A = A(q) ; % overwrite
> clear q ;
>
> If this is not an option, you can turn to very inefficient loops, like this one, which does not occupy (a lot of) additional memory ...
>
> A = rand(10,1) ;
> idx = 1 ;
> while (idx <= numel(A)),
> if A(idx) > 0.5,
> idx = idx + 1 ; % goto next element
> else
> A(idx) = [] ; % remove element from list
> end
> end
>
> hth
> Jos

Does enybody know what really happens in this kind of loop?
Each time when an element is removed (assigned to empty), does MATLAB have to find a new contiguous memory block for the new slightly smaller array? Thus fragmenting memory.

Re: filtering data problem

Mike — Fri, 24 Apr 2009 04:09:23 -0400

On Apr 23, 2:30=A0pm, "Jos " <#10...@fileexchange.com> wrote:
> "Mike " <sulfate...@gmail.com> wrote in message <gsoc01$eo...@fred.mathwo=
rks.com>...
> > Hi
> > =A0 =A0 I usually need to filter data. =A0Sometimes I use logical-index=
ing, but I found there are many temporary arrays I created. =A0Does that wa=
y waste memory? =A0How to avoid this?
>
> > Mike
>
> Logical indexing is very powerful but indeed creates a (temporary) array.=
However, they use 1/8 of the memory occupied by =A0double array of the sam=
e size:
> A =3D rand(100,1) ; B =3D A > 0.5 ; whos
>
> You can clear out, or overwrite, any variables you no longer need:
> A =3D rand(10,1) ;
> q =3D A > 0.5 ;
> A =3D A(q) ; % overwrite
> clear q ;
>
> If this is not an option, you can turn to very inefficient loops, like th=
is one, which does not occupy (a lot of) additional memory ...
>
> A =3D rand(10,1) ;
> idx =3D 1 ;
> while (idx <=3D numel(A)),
> =A0 if A(idx) > 0.5,
> =A0 =A0 idx =3D idx + 1 ; % goto next element
> =A0 else
> =A0 =A0 A(idx) =3D [] ; % remove element from list
> =A0 end
> end
>
> hth
> Jos

Thank you very much, Jos.
Allow me to add some cputime elapse time statements.

N=3D100000;
t =3D cputime;
A =3D rand(N,1) ;
q =3D A > 0.5 ;
A =3D A(q) ; % overwrite
clear q ;
e=3Dcputime-t

t =3D cputime;
A =3D rand(N,1) ;
idx =3D 1 ;
while (idx <=3D numel(A)),
  if A(idx) > 0.5,
    idx =3D idx + 1 ; % goto next element
  else
    A(idx) =3D [] ; % remove element from list
  end
end
e=3Dcputime-t

e =3D

    0.0156

e =3D

   46.5781

The latter one is very inefficient as you say.
In practice, I filter data using the first way continuously.
The inefficiency would propagate one by one using element-by-element
way.

Mike

Re: filtering data problem

Bruno Luong — Fri, 24 Apr 2009 06:40:22 -0400

"Pekka Kumpulainen" <pekka.nospam.kumpulainen@tut.please.fi> wrote in message <gsp87p$cdu$1@fred.mathworks.com>...
> "Jos " <#10584@fileexchange.com> wrote in message <gsp1tf$bno$1@fred.mathworks.com>...

>
> Does enybody know what really happens in this kind of loop?
> Each time when an element is removed (assigned to empty), does MATLAB have to find a new contiguous memory block for the new slightly smaller array? Thus fragmenting memory.

Yes it does (Using my own PrintPtr mex command to track data pointer):

>> a=[1:10]
a =
1 2 3 4 5 6 7 8 9 10

>> PrintPtr(a)
Ptr=0E23B720x

>> a(1)=[]
a =
2 3 4 5 6 7 8 9 10

>> PrintPtr(a)
Ptr=0E3F1790x % <- it changes

Bruno

Re: filtering data problem

Pekka Kumpulainen — Fri, 24 Apr 2009 07:09:03 -0400

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <gsrmsm$5pv$1@fred.mathworks.com>...
> "Pekka Kumpulainen" <pekka.nospam.kumpulainen@tut.please.fi> wrote in message <gsp87p$cdu$1@fred.mathworks.com>...
> > "Jos " <#10584@fileexchange.com> wrote in message <gsp1tf$bno$1@fred.mathworks.com>...
>
> >
> > Does enybody know what really happens in this kind of loop?
> > Each time when an element is removed (assigned to empty), does MATLAB have to find a new contiguous memory block for the new slightly smaller array? Thus fragmenting memory.
>
> Yes it does (Using my own PrintPtr mex command to track data pointer):
>
> >> a=[1:10]
> a =
> 1 2 3 4 5 6 7 8 9 10
>
> >> PrintPtr(a)
> Ptr=0E23B720x
>
> >> a(1)=[]
> a =
> 2 3 4 5 6 7 8 9 10
>
> >> PrintPtr(a)
> Ptr=0E3F1790x % <- it changes
>
> Bruno

Just what I thought, thanks.
Would that PrintPtr be available? It would be handy sometimes to see what actually happens in the memory.

Re: filtering data problem

Pekka Kumpulainen — Fri, 24 Apr 2009 07:16:03 -0400

"Pekka Kumpulainen" <pekka.nospam.kumpulainen@tut.please.fi> wrote in message <gsroif$dt$1@fred.mathworks.com>...
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <gsrmsm$5pv$1@fred.mathworks.com>...
> > "Pekka Kumpulainen" <pekka.nospam.kumpulainen@tut.please.fi> wrote in message <gsp87p$cdu$1@fred.mathworks.com>...
> > > "Jos " <#10584@fileexchange.com> wrote in message <gsp1tf$bno$1@fred.mathworks.com>...
> >
> > >
> > > Does enybody know what really happens in this kind of loop?
> > > Each time when an element is removed (assigned to empty), does MATLAB have to find a new contiguous memory block for the new slightly smaller array? Thus fragmenting memory.
> >
> > Yes it does (Using my own PrintPtr mex command to track data pointer):
> >
> > >> a=[1:10]
> > a =
> > 1 2 3 4 5 6 7 8 9 10
> >
> > >> PrintPtr(a)
> > Ptr=0E23B720x
> >
> > >> a(1)=[]
> > a =
> > 2 3 4 5 6 7 8 9 10
> >
> > >> PrintPtr(a)
> > Ptr=0E3F1790x % <- it changes
> >
> > Bruno
>
> Just what I thought, thanks.
> Would that PrintPtr be available? It would be handy sometimes to see what actually happens in the memory.

Forget the previous. I only serched FEX before posting. Found it two minutes after that.