Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Simultaneous disk read and computations
Date: Sun, 31 May 2009 20:17:01 +0000 (UTC)
Organization: Cornell University
Lines: 23
Message-ID: <gvuojt$msr$1@fred.mathworks.com>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-03-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1243801021 23451 172.30.248.38 (31 May 2009 20:17:01 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Sun, 31 May 2009 20:17:01 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 1327593
Xref: news.mathworks.com comp.soft-sys.matlab:543823


Hello,
I have amount of data stored on disk - now about 10GB stored as 20 different files in 500MB blocks.  I need to loop through each block, do some computations, take in the next block etc.  Right now, reading the file (binary format) takes about as much time as my computations.  I was wondering if it were possible to read in the next block while the CPU is processing the first block.
Code looks something like this:

for i=1:nBlock
A=fread(fid{i},inf,'*uint8');
result{i} = doSomething(A);
end

I would like to do something like this:

A{1} = fread(fid{1},inf,'*uint8');
for i=1:nBlock-1
   indCurr = mod(i-1,2)+1; %1 if i if odd, 2 if i is even
   indNext = mod(i,2)+1; %1 if i is even, 2 if i is odd
   b{i} = doSomething(A{indCurr});             % CPU intensive
   A{indNext} = fread(fid{1+1},inf,'*uint8'); % disk intensive
end
b{nBlock} = doSomthing(A{indNext});

where the lines labeled "CPU intensive" and "disk intensive" are done in parralel.
Thanks,
Nick