Path: news.mathworks.com!not-for-mail
From: "jay vaughan" <jvaughan5.nospam@gmail.com>
Newsgroups: comp.soft-sys.matlab
Subject: Re: loading in large data...
Date: Wed, 7 May 2008 04:07:02 +0000 (UTC)
Organization: harvard
Lines: 62
Message-ID: <fvr9t6$67f$1@fred.mathworks.com>
References: <fvod1b$di3$1@fred.mathworks.com>
Reply-To: "jay vaughan" <jvaughan5.nospam@gmail.com>
NNTP-Posting-Host: webapp-05-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1210133222 6383 172.30.248.35 (7 May 2008 04:07:02 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Wed, 7 May 2008 04:07:02 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 1215048
Xref: news.mathworks.com comp.soft-sys.matlab:467066


"jay vaughan" <jvaughan5.nospam@gmail.com> wrote in message
<fvod1b$di3$1@fred.mathworks.com>...
> Hi,
> 
> I am having some trouble optimizing the loading & handling
> of large files (movies, in a kind of weird format). Any
> comment on the following?
> 
> 1) My code for loading the data (see below) was slow, and it
> didn't scale linearly with the number of iterations of the
> loop as I had thought it would. Any ideas on how to speed
it up?
> 
> num_iterations = [10 30 100 300]; % # of frames loaded
> time_measured = [0.098 0.37 2.18 14.3]; % time in seconds
> 
> 2) Loading in the data as uint8. I think I want to work with
> the data as uint8 not double since the data is 8 bit anyway
> and uint8 is 8x more compact than double. Does this seem
> like a good strategy?
> 
> Below is my code to load the data. The details of the file
> format are after the code in case it is helpful.
> 
> [fid,msga]=fopen(filename,'r','ieee-le');
> 
> % first find xy dimensions
> x_dim = fread(fid,1,'uint16');
> y_dim = fread(fid,1,'uint16');
> 
> % loop through frames reading the data, reading first the 
> % frame number then the frame data
> frame_num = fread(fid,1,'uint16');
> mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );
> for k = 2:100;
>     frame_num = fread(fid,1,'uint16');
>     frame = fread(fid,[x_dim, y_dim],'uint8');
>     mov = cat(3,mov,uint8(frame));
> end
> 
> fclose(fid);
> 
> 
> The movie is format is as follows. First, 4 bytes indicate
> the x dimension size and y dimension size (2 bytes each).
> Then two bytes indicate a frame number, followed by a single
> byte for each of x_dim*y_dim pixels to form a frame. The
> frame number, frame data structure is repeated until the end
> of the movie. 


Peter and John,

thanks for the replies. Preallocation helped, as did keeping
the frame in uint8.

Now I am working on how to display the movie frames quickly
in a GUI when controlled with arrow keys. We'll see how that
goes...


J