Path: news.mathworks.com!newsfeed-00.mathworks.com!panix!bloom-beacon.mit.edu!llnews!53ab2750!not-for-mail
From: Peter Boettcher <boettcher@ll.mit.edu>
Newsgroups: comp.soft-sys.matlab
Subject: Re: loading in large data...
References: <fvod1b$di3$1@fred.mathworks.com>
Message-ID: <muyhcdbiopa.fsf@G99-Boettcher.llan.ll.mit.edu>
Organization: MIT Lincoln Laboratory
User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/23.0.0 (gnu/linux)
Cancel-Lock: sha1:t3Il5ZhoofI0EcxU1eUwVC+l5lM=
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Lines: 62
Date: Tue, 06 May 2008 08:47:45 -0400
NNTP-Posting-Host: 155.34.163.114
X-Complaints-To: news@ll.mit.edu
X-Trace: llnews 1210077425 155.34.163.114 (Tue, 06 May 2008 08:37:05 EDT)
NNTP-Posting-Date: Tue, 06 May 2008 08:37:05 EDT
Xref: news.mathworks.com comp.soft-sys.matlab:466890


"jay vaughan" <jvaughan5.nospam@gmail.com> writes:

> Hi,
>
> I am having some trouble optimizing the loading & handling
> of large files (movies, in a kind of weird format). Any
> comment on the following?
>
> 1) My code for loading the data (see below) was slow, and it
> didn't scale linearly with the number of iterations of the
> loop as I had thought it would. Any ideas on how to speed it up?
>
> num_iterations = [10 30 100 300]; % # of frames loaded
> time_measured = [0.098 0.37 2.18 14.3]; % time in seconds
>
> 2) Loading in the data as uint8. I think I want to work with
> the data as uint8 not double since the data is 8 bit anyway
> and uint8 is 8x more compact than double. Does this seem
> like a good strategy?
>
> Below is my code to load the data. The details of the file
> format are after the code in case it is helpful.
>
> [fid,msga]=fopen(filename,'r','ieee-le');
>
> % first find xy dimensions
> x_dim = fread(fid,1,'uint16');
> y_dim = fread(fid,1,'uint16');
>
> % loop through frames reading the data, reading first the 
> % frame number then the frame data
> frame_num = fread(fid,1,'uint16');
> mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );

Instead of fread converting to double, then uint8() converting back to
uint8, specify the fread format as '*uint8', to keep the data in it's
original format.

> for k = 2:100;
>     frame_num = fread(fid,1,'uint16');
>     frame = fread(fid,[x_dim, y_dim],'uint8');
>     mov = cat(3,mov,uint8(frame));

As John says, dynamically expanding the array like this requires a
memory allocation and memcpy for each frame.  

> end
>
> fclose(fid);

Instead, preallocate your array then fill it in:

mov = zeros(x_dim, y_dim, 100, 'uint8');

for k=1:100
  frame_num = fread(fid, 1, 'uint16');
  mov(:,:,k) = fread(fid, [x_dim y_dim], '*uint8');
end



-Peter