|
"James Tursa" <aclassyguywithaknotac@hotmail.com> wrote in message <gr76gq$5ap$1@fred.mathworks.com>...
> "Derek O'Connor" <derekroconnor@eircom.net> wrote in message <gr6cci$k50$1@fred.mathworks.com>...
> >
> > 3. You say "in the C = operations(C) instruction you need a copy of C". Are you sure of that? Can anyone verify that statement? Can you explain these lines of code?
> >
> > >> C = 2*C; % --- out-of-memory failure ---
> > >> C = C+C; % --- out-of-memory failure ---
> > >>for i = 1:n, for j = 1:n, C(i,j) = 2*C(i,j);end;end; % --- works ok ---
> > >>for i = 1:n, for j = 1:n, C(i,j) = C(i,j)+C(i,j);end;end; % --- works ok ---
> >
>
> MATLAB first creates the result of 2*C. This requires a matrix the same size of C, which is what the phrase "copy of C" actually meant. So that doubles the amount of storage needed, at least temporarily. Then, had it worked, the original C memory is freed and C is set to this new result. That is why your double loops do not run out of memory ... they do the operation in place. Same comments for the C+C equation and loop. The fact that your 2*C operation runs out of memory whereas your loops do not should be proof enough that a temp matrix the same size as C is used.
>
> If you really need to do these simple operations on a very large matrix that is not sharing memory with another variable, you can do them inplace in a mex routine. e.g.
>
> #include "mex.h"
> void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
> {
> mwSize i, numel;
> double *pr;
> if( nrhs != 1 || nlhs != 0 ) {
> mexErrMsgTxt("Need one input and no outputs.");
> }
> if( !mxIsDouble(prhs[0]) ) {
> mexErrMsgTxt("Input must be double.");
> }
> if( mxIsSparse(prhs[0]) ) {
> numel = *(mxGetJc(prhs[0])+ mxGetN(prhs[0]));
> } else {
> numel = mxGetNumberOfElements(prhs[0]);
> }
> pr = mxGetPr(prhs[0]);
> for( i=0; i<numel; i++ ) {
> pr[i] *= 2.0;
> }
> }
>
> To mex it just call the file mult2.c and then do this:
>
> >> mex mult2.c
>
> Here is a timing test of the three methods:
>
> >> a=rand(5000);
> >> tic;2*a;toc
> Elapsed time is 0.458821 seconds.
> >> tic;for k=1:5000;for m=1:5000; a(k,m) = 2*a(k,m);end;end;toc
> Elapsed time is 43.492720 seconds.
> >> tic;mult2(a);toc
> Elapsed time is 0.134097 seconds.
>
> > 5. "... and last, you really need a double random number? Try to recast to something less expensive."
> >
> > Oh dear! I've been here before. See James Tursa's and my posts on this subject here:
> > http://www.mathworks.com/matlabcentral/newsreader/view_thread/239186#637344
> >
>
> Based on that thread, I have been experimenting with a sparse int8 class that I have created. So far it just does simple things like plus, minus, and times. I am slowly working on adding more capability for potential upload to the FEX someday. Would this be of interest to you? If so, what operations would you need/want for a sparse int8 class?
>
> James Tursa
============================================
Dear James,
I'll look at your mex functions later but respond to the first part of
your post here.
You say :
"MATLAB first creates the result of 2*C. This requires a matrix the
same size of C, which is what the phrase "copy of C" actually meant.
So that doubles the amount of storage needed, at least temporarily."
Your statement seems to be verified by this result :
>> clear all; n = 3.0*10^4; C = 2*rand(n,n);
??? Out of memory. Type HELP MEMORY for your options.
But it seems to be contradicted by this result :
>> clear all; n = 3.0*10^4; C = rand(n,n); C = C+C;
>> clear all; n = 3.0*10^4; C = rand(n,n); C = 2*C;
>> clear all; n = 4.2*10^4; C = rand(n,n); C = 2*C;
>> memory
Maximum possible array: 214 MB (2.249e+008 bytes) *
Memory available for all arrays: 214 MB (2.249e+008 bytes) *
Memory used by MATLAB: 13837 MB (1.451e+010 bytes)
Physical Memory (RAM): 16381 MB (1.718e+010 bytes)
WTM (Windows Task Manager) reports 15.5 GB (96%) physical memory in use.
It is very important when problem sizes are close to the memory limit that
no unecessary temporaries are generated. I wrote the small function below
to show that you can solve problems that are close to the memory limit and
that use very little extra space above that for the problem.
You can see that it solved a 42,000 x 42,000 dense linear equations problem
in core using 96% of the physical memory. Notice that it uses the two-step
trick shown above : C = rand(n,n); C = C/n; That is bad.
Programmers should not use tricks.
Regards,
Derek O'Connor
>-------------------------------------------------------------------------------
function [C,d,xsol,xnew,res] = TestFxPtIter(n,tol,maxits);
%
% Generate nxn matrix: C, nx1 vectors: d,xsol, to test
% Iteration x(k+1) = Cx(k)+d : converges if ||C|| < 1;
% Set tol = 0 for full prec
% USE : [C,d,xsol,xnew,res] = TestFxPtIter(10^4,0,100);
%
% Derek O'Connor, 4 April 2009. derekroconnor@eircom.net
% --------- Generate the problem data C and d -----------------
C = rand(n,n); % Matlab gives out-of-mem error
C = C/n; % if C = rand(n,n)/n is used.
xsol = ones(n,1); % any given solution will do
d = xsol-C*xsol; %
xold = zeros(n,1); % In theory any starting value will do
%
% --------- Start of iterations -------------------------------
tic;
converged = false; k = 0;
while ~converged && k <= maxits
k = k+1;
xnew = C*xold+d; % O(n^2) work per iteration
nreldelx = norm(xnew-xold,inf)/norm(xold,inf);
converged = nreldelx <= tol+eps;
xold = xnew;
end;
res = [n k nrdelx toc];
%-------------------- end of TestFxPtIter ---------------------
% clear all;
% [C,d,xsol,xnew,res] = TestFxPtIter(4.2*10^4,10^(-7),100);
% res =
% n = 42000 iters = 24 tol = 6.0267e-008 time = 118.36 secs
% 2 x Quad Xeon 5345s @ 2.33GHz, 16GB ram. Matlab 7.6, Vista 64 bit.
% WTN : 15.5 GB, 96% Phys Mem used in 87 processes
|
Feed for this Thread