Path: news.mathworks.com!not-for-mail
From: "John D'Errico" <woodchips@rochester.rr.com>
Newsgroups: comp.soft-sys.matlab
Subject: Re: loading in large data...
Date: Tue, 6 May 2008 02:36:03 +0000 (UTC)
Organization: John D'Errico (1-3LEW5R)
Lines: 37
Message-ID: <fvog6j$hl4$1@fred.mathworks.com>
References: <fvod1b$di3$1@fred.mathworks.com>
Reply-To: "John D'Errico" <woodchips@rochester.rr.com>
NNTP-Posting-Host: webapp-02-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1210041363 18084 172.30.248.37 (6 May 2008 02:36:03 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Tue, 6 May 2008 02:36:03 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 869215
Xref: news.mathworks.com comp.soft-sys.matlab:466817


"jay vaughan" <jvaughan5.nospam@gmail.com> wrote in message 
<fvod1b$di3$1@fred.mathworks.com>...
> Hi,
> 
> I am having some trouble optimizing the loading & handling
> of large files (movies, in a kind of weird format). Any
> comment on the following?
> 
> 1) My code for loading the data (see below) was slow, and it
> didn't scale linearly with the number of iterations of the
> loop as I had thought it would. Any ideas on how to speed it up?

Its funny. But I'd never have expected code
that does not pre-allocate an array, then
repeatedly concatenates data to the array
to behave in a linear fashion.

Your test does exactly what you should
NEVER do. When you append data to an
array, Matlab must dynamically re-allocate
that array with every iteration of the loop.
This is a trivial thing when you do it once,
or on a tiny array of numbers, just a few
times.

When you do that same operation on a
large array, and then do it several hundred
times, it takes much time. If the array is
large enough, this can potentially cause
seriously bad fragmentation of your
memory. It can force Matlab to begin
swapping large blocks of data in and out
of virtual memory. The point is, this is
NOT at all a process that is linear in the
time required.

John