"Qizhu " <qizhu.li@magd.ox.ac.uk> wrote in message
news:lsijcm$bdh$1@newscl01ah.mathworks.com...
> "Steven Lord" <Steven_Lord@mathworks.com> wrote in message
> <lsihao$5eh$1@newscl01ah.mathworks.com>...
*snip*
> Thanks Steve for the reply! Yes I am aware of the matfile command. In fact
> that's what I am planning to use after I manage (if ever..) to create this
> huge matrix, so that I can load just part of the matrix each time.
> What I am trying to do with this matrix to perform probabilistic PCA. In
> fact the for loop I described previously is just the first step: to
> calculate the mean along each row of the raw data matrix (which comes in
> uint8 format), subtract the mean off each entry, and save the centred data
> as a new mat file with single precision. Because of the huge dimension of
> the raw data matrix (130k by 200k ) , I can't do the simply oneliner:
> mean(data,2). I have to read in a few rows, find the means, centre them,
> save, and then process the next few rows.
If you have a parallel cluster of machines, creating that large matrix as a
distributed array may help you. In that case you may in fact be able to use
that simple oneliner, letting Parallel Computing Toolbox handle performing
the computations in a distributed manner.
http://www.mathworks.com/help/distcomp/distributedarraysandspmd.html
> Your answer seem to suggest that it is not possible to append to existing
> variables by concatenating extra matrix at the end. Did I get this right?
Not using "save append", no. In memory, yes. Via a MATFILE object, not by
concatenation but by indexed assignment.
> Also, is it possible to create an EMPTY single precision v7.3 mat file on
> hard disc without creating it in RAM, so that I can use the matfile
> function to partially load it and save my centred matrix portion into the
> right position in this matrix?
Answering the question you meant to ask, you can create a single precision
in the MATfile without running into memory problems, then expand it via a
MATFILE object. You _would_ need to create single precision scalars in
memory, but if those cause memory problems lots of other programs on your
machine are probably in trouble as well as MATLAB.
>> m = matfile('expandingSingleMatrix.mat', 'Writable', true);
>> m.s = single(1);
>> whos file expandingSingleMatrix.mat
Name Size Bytes Class Attributes
s 1x1 4 single
>> m.s(1000, 1000) = single(pi);
>> whos file expandingSingleMatrix.mat
Name Size Bytes Class Attributes
s 1000x1000 4000000 single
After I created the variable s, it was scalar. When I expanded it by
assigning a value to an element that didn't already exist, it grew.
Answering the question you _actually_ asked, creating an empty single
precision array will not cause ANY problem with memory. By definition, an
empty array has at least one dimension of size 0, meaning it has no elements
and takes up no memory.
>> tic; z = ones(1e14, 1e14, 0, 'single'); toc
Elapsed time is 0.000228 seconds.
>> whos z
Name Size Bytes Class
Attributes
z 100000000000000x100000000000000x0 0 single
Now if you were to try to expand THIS matrix, you WOULD have a memory
problem.

Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com
