Path: news.mathworks.com!not-for-mail
From: "jay vaughan" <jvaughan5.nospam@gmail.com>
Newsgroups: comp.soft-sys.matlab
Subject: loading in large data...
Date: Tue, 6 May 2008 01:42:03 +0000 (UTC)
Organization: harvard
Lines: 46
Message-ID: <fvod1b$di3$1@fred.mathworks.com>
Reply-To: "jay vaughan" <jvaughan5.nospam@gmail.com>
NNTP-Posting-Host: webapp-05-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1210038123 13891 172.30.248.35 (6 May 2008 01:42:03 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Tue, 6 May 2008 01:42:03 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 1215048
Xref: news.mathworks.com comp.soft-sys.matlab:466815



Hi,

I am having some trouble optimizing the loading & handling
of large files (movies, in a kind of weird format). Any
comment on the following?

1) My code for loading the data (see below) was slow, and it
didn't scale linearly with the number of iterations of the
loop as I had thought it would. Any ideas on how to speed it up?

num_iterations = [10 30 100 300]; % # of frames loaded
time_measured = [0.098 0.37 2.18 14.3]; % time in seconds

2) Loading in the data as uint8. I think I want to work with
the data as uint8 not double since the data is 8 bit anyway
and uint8 is 8x more compact than double. Does this seem
like a good strategy?

Below is my code to load the data. The details of the file
format are after the code in case it is helpful.

[fid,msga]=fopen(filename,'r','ieee-le');

% first find xy dimensions
x_dim = fread(fid,1,'uint16');
y_dim = fread(fid,1,'uint16');

% loop through frames reading the data, reading first the 
% frame number then the frame data
frame_num = fread(fid,1,'uint16');
mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );
for k = 2:100;
    frame_num = fread(fid,1,'uint16');
    frame = fread(fid,[x_dim, y_dim],'uint8');
    mov = cat(3,mov,uint8(frame));
end

fclose(fid);


The movie is format is as follows. First, 4 bytes indicate
the x dimension size and y dimension size (2 bytes each).
Then two bytes indicate a frame number, followed by a single
byte for each of x_dim*y_dim pixels to form a frame. The
frame number, frame data structure is repeated until the end
of the movie.