Path: news.mathworks.com!not-for-mail From: "heidi jiang" <heydee@gmail.com> Newsgroups: comp.soft-sys.matlab Subject: Re: faster way to implement corr Date: Wed, 16 May 2012 17:06:15 +0000 (UTC) Organization: The MathWorks, Inc. Lines: 26 Message-ID: <jp0mq7$snn$1@newscl01ah.mathworks.com> References: <jourtj$quo$1@newscl01ah.mathworks.com> <jov0c3$ebk$1@newscl01ah.mathworks.com> <jov1c9$i10$1@newscl01ah.mathworks.com> <jov443$snv$1@newscl01ah.mathworks.com> <jovb54$q02$1@newscl01ah.mathworks.com> Reply-To: "heidi jiang" <heydee@gmail.com> NNTP-Posting-Host: www-02-blr.mathworks.com Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Trace: newscl01ah.mathworks.com 1337187975 29431 172.30.248.47 (16 May 2012 17:06:15 GMT) X-Complaints-To: news@mathworks.com NNTP-Posting-Date: Wed, 16 May 2012 17:06:15 +0000 (UTC) X-Newsreader: MATLAB Central Newsreader 3488832 Xref: news.mathworks.com comp.soft-sys.matlab:768052 "Roger Stafford" wrote in message <jovb54$q02$1@newscl01ah.mathworks.com>... > "Roger Stafford" wrote in message <jov443$snv$1@newscl01ah.mathworks.com>... > > This could be done with a large set of vectors as a matrix multiplication, but I suspect you would have memory problems with a 230000 by 230000 matrix, and would at least have to do such a multiplication piecemeal. > - - - - - - - - - > There is one point I want to correct in my previous discussion. I mentioned a 230000 by 230000 matrix, and that is surely incorrect. Each vector undoubtedly has far fewer than 230000 elements. > > If each of your vectors has n elements, then let A be a 230000 by n matrix in which the rows are the modified vectors and for which you hopefully have enough memory. Then I envision an iteration that takes m of the vectors at a time: > > for k = 1:230000/m % m a factor of 230000 > C = A*A((k-1)*m+1:k*m,:).'; % <-- Correlations > % C will be 230000 by m in size > % Each column has correlations of all vectors w.r. to > % one of the vectors (k-1)*m+1 to k*m. > % Apply nonlinear ops. on each column of C to get "average" correlation > % Result is m "average" correlations which should be stored > end > > Here you would make m as large as possible and still be able to store a 230000 by m matrix C. Each trip through the for-loop would encompass what m of your iterations accomplished and 'corr' will not have been called. > > I hope all this makes sense. > > Roger Stafford Hi, This makes sense and I'll try it. Thank you!