MATLAB Central Newsreader - loading in large data...

loading in large data...

jay vaughan — Tue, 06 May 2008 01:42:03 -0400

Hi,

I am having some trouble optimizing the loading & handling
of large files (movies, in a kind of weird format). Any
comment on the following?

1) My code for loading the data (see below) was slow, and it
didn't scale linearly with the number of iterations of the
loop as I had thought it would. Any ideas on how to speed it up?

num_iterations = [10 30 100 300]; % # of frames loaded
time_measured = [0.098 0.37 2.18 14.3]; % time in seconds

2) Loading in the data as uint8. I think I want to work with
the data as uint8 not double since the data is 8 bit anyway
and uint8 is 8x more compact than double. Does this seem
like a good strategy?

Below is my code to load the data. The details of the file
format are after the code in case it is helpful.

[fid,msga]=fopen(filename,'r','ieee-le');

% first find xy dimensions
x_dim = fread(fid,1,'uint16');
y_dim = fread(fid,1,'uint16');

% loop through frames reading the data, reading first the
% frame number then the frame data
frame_num = fread(fid,1,'uint16');
mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );
for k = 2:100;
    frame_num = fread(fid,1,'uint16');
    frame = fread(fid,[x_dim, y_dim],'uint8');
    mov = cat(3,mov,uint8(frame));
end

fclose(fid);

The movie is format is as follows. First, 4 bytes indicate
the x dimension size and y dimension size (2 bytes each).
Then two bytes indicate a frame number, followed by a single
byte for each of x_dim*y_dim pixels to form a frame. The
frame number, frame data structure is repeated until the end
of the movie.

Re: loading in large data...

John D'Errico — Tue, 06 May 2008 02:36:03 -0400

Re: loading in large data...

Peter Boettcher — Tue, 06 May 2008 12:47:45 -0400

"jay vaughan" <jvaughan5.nospam@gmail.com> writes:

> Hi,
>
> I am having some trouble optimizing the loading & handling
> of large files (movies, in a kind of weird format). Any
> comment on the following?
>
> 1) My code for loading the data (see below) was slow, and it
> didn't scale linearly with the number of iterations of the
> loop as I had thought it would. Any ideas on how to speed it up?
>
> num_iterations = [10 30 100 300]; % # of frames loaded
> time_measured = [0.098 0.37 2.18 14.3]; % time in seconds
>
> 2) Loading in the data as uint8. I think I want to work with
> the data as uint8 not double since the data is 8 bit anyway
> and uint8 is 8x more compact than double. Does this seem
> like a good strategy?
>
> Below is my code to load the data. The details of the file
> format are after the code in case it is helpful.
>
> [fid,msga]=fopen(filename,'r','ieee-le');
>
> % first find xy dimensions
> x_dim = fread(fid,1,'uint16');
> y_dim = fread(fid,1,'uint16');
>
> % loop through frames reading the data, reading first the
> % frame number then the frame data
> frame_num = fread(fid,1,'uint16');
> mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );

Instead of fread converting to double, then uint8() converting back to
uint8, specify the fread format as '*uint8', to keep the data in it's
original format.

> for k = 2:100;
> frame_num = fread(fid,1,'uint16');
> frame = fread(fid,[x_dim, y_dim],'uint8');
> mov = cat(3,mov,uint8(frame));

As John says, dynamically expanding the array like this requires a
memory allocation and memcpy for each frame.

> end
>
> fclose(fid);

Instead, preallocate your array then fill it in:

mov = zeros(x_dim, y_dim, 100, 'uint8');

for k=1:100
frame_num = fread(fid, 1, 'uint16');
mov(:,:,k) = fread(fid, [x_dim y_dim], '*uint8');
end

-Peter

Re: loading in large data...

jay vaughan — Wed, 07 May 2008 04:07:02 -0400

"jay vaughan" <jvaughan5.nospam@gmail.com> wrote in message
<fvod1b$di3$1@fred.mathworks.com>...
> Hi,
>
> I am having some trouble optimizing the loading & handling
> of large files (movies, in a kind of weird format). Any
> comment on the following?
>
> 1) My code for loading the data (see below) was slow, and it
> didn't scale linearly with the number of iterations of the
> loop as I had thought it would. Any ideas on how to speed
it up?
>
> num_iterations = [10 30 100 300]; % # of frames loaded
> time_measured = [0.098 0.37 2.18 14.3]; % time in seconds
>
> 2) Loading in the data as uint8. I think I want to work with
> the data as uint8 not double since the data is 8 bit anyway
> and uint8 is 8x more compact than double. Does this seem
> like a good strategy?
>
> Below is my code to load the data. The details of the file
> format are after the code in case it is helpful.
>
> [fid,msga]=fopen(filename,'r','ieee-le');
>
> % first find xy dimensions
> x_dim = fread(fid,1,'uint16');
> y_dim = fread(fid,1,'uint16');
>
> % loop through frames reading the data, reading first the
> % frame number then the frame data
> frame_num = fread(fid,1,'uint16');
> mov = uint8( fread(fid,[x_dim, y_dim],'uint8') );
> for k = 2:100;
> frame_num = fread(fid,1,'uint16');
> frame = fread(fid,[x_dim, y_dim],'uint8');
> mov = cat(3,mov,uint8(frame));
> end
>
> fclose(fid);
>
>
> The movie is format is as follows. First, 4 bytes indicate
> the x dimension size and y dimension size (2 bytes each).
> Then two bytes indicate a frame number, followed by a single
> byte for each of x_dim*y_dim pixels to form a frame. The
> frame number, frame data structure is repeated until the end
> of the movie.

Peter and John,

thanks for the replies. Preallocation helped, as did keeping
the frame in uint8.

Now I am working on how to display the movie frames quickly
in a GUI when controlled with arrow keys. We'll see how that
goes...

J