Path: news.mathworks.com!newsfeed-00.mathworks.com!news.kjsl.com!feeder.erje.net!eu.feeder.erje.net!newsfeed.datemas.de!rt.uk.eu.org!aioe.org!.POSTED!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Re: very large array
Date: Mon, 24 Jun 2013 11:13:24 -0500
Organization: Aioe.org NNTP Server
Lines: 59
Message-ID: <kq9r5n$21t$1@speranza.aioe.org>
References: <kq9hdf$bsa$1@newscl01ah.mathworks.com> <kq9khs$kvi$1@newscl01ah.mathworks.com> <kq9n1f$smu$1@newscl01ah.mathworks.com> <kq9nib$12h$1@newscl01ah.mathworks.com>
NNTP-Posting-Host: jy+QfeYoQFMk8Or4YOj/xQ.user.speranza.aioe.org
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Complaints-To: abuse@aioe.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20
X-Notice: Filtered by postfilter v. 0.8.2
Xref: news.mathworks.com comp.soft-sys.matlab:798117

On 6/24/2013 10:11 AM, Lorenzo Quadri wrote:
> "Lorenzo Quadri" <quadrilo_sub_r@gmail.com> wrote in message
> <kq9n1f$smu$1@newscl01ah.mathworks.com>...
>> "Steven_Lord" <slord@mathworks.com> wrote in message > > Not only will
>> this be slow, it will also error. If you have a 10 row array:
...
>> you'd walk off the end of the array if any of the rows are deleted.
>...

>> I replaced the for loop with a while one like:
>>
>> a=length(dati)
>> while (a>0)
>>   if((sum(dati(i,:))<355) & range(dati(i,:))>20)
>>     a = a - 1;
>>     dati(i,:) = [];
>>   end
>> end
>>
>> I'll try evaluate it with logical indexing as you suggested.
>> Copy the array in an other one take about 1,5 hrs every 1000000
>> iterations
>> so the whole operation is exstimated in about 300 hrs or 12,5 days
>> (too much, conditions change meanwhile).
>>
>>
>> thank you very much
>
> ops sorry
>
> while (a>0)
>   if((sum(dati(i,:))<355) & range(dati(i,:))>20)
>     dati(i,:) = [];
>   end
>   a = a - 1;
> end

Now you're not incrementing i so you'll process the same row over and 
over and over...

While doing it by a loop is _NOT_ the way for large cases, sometimes it 
is handy and time isn't an issue for small array sizes.  The way in 
general to do such things is to start at the end and progress 
forwards--that way the lower indices aren't affected by the deleted rows...

for i=length(dat):-1:1
   if((sum(dati(i,:))<355) & range(dati(i,:))>20)
     dati(i,:) = [];
   end
end

Now it won't run off the end (but it will still run a _loooong_ time, 
methinks...

See other comments on "why" and perhaps a more-quicker way of breaking 
it into chunks if you're causing memory paging w/ it in one full array 
in (virtual) memory.

--