Simultaneous disk read and computations

NIcholas — Sun, 31 May 2009 20:17:01 -0400

Hello,
I have amount of data stored on disk - now about 10GB stored as 20 different files in 500MB blocks. I need to loop through each block, do some computations, take in the next block etc. Right now, reading the file (binary format) takes about as much time as my computations. I was wondering if it were possible to read in the next block while the CPU is processing the first block.
Code looks something like this:

for i=1:nBlock
A=fread(fid{i},inf,'*uint8');
result{i} = doSomething(A);
end

I would like to do something like this:

A{1} = fread(fid{1},inf,'*uint8');
for i=1:nBlock-1
   indCurr = mod(i-1,2)+1; %1 if i if odd, 2 if i is even
   indNext = mod(i,2)+1; %1 if i is even, 2 if i is odd
   b{i} = doSomething(A{indCurr}); % CPU intensive
   A{indNext} = fread(fid{1+1},inf,'*uint8'); % disk intensive
end
b{nBlock} = doSomthing(A{indNext});

where the lines labeled "CPU intensive" and "disk intensive" are done in parralel.
Thanks,
Nick

MATLAB Central Newsreader - Simultaneous disk read and computations

Simultaneous disk read and computations