Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Using xcorr on a 10512 x 10512 with 6 lags

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: James

Date: 8 Jul, 2010 16:04:06

Message: 1 of 7

Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6 shifts. The result is 13 x 10512^2 matrix which it seems kills my computer. I'm using a MAC with 32GB of RAM and it's still crashing! I was wondering if there was a way to reduce the memory imprint of xcorr? The dataset is a time-series and I'm trying to get the cross correlation between each row with 6 month lags.
The only other solution that I found was to use a nested for-loop:
for i = 1:10512
     for j = 1: 10512
              cc(:,i) = xcorr(data(i,:),data(j,:),6,'coeff');
      end
end

But this is so slow it takes more than a day to get the cc matrix! Is there another way I could perform row-wise cross correlation without the nested for-loop?
Thank you.
James

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: Steven Lord

Date: 8 Jul, 2010 18:03:17

Message: 2 of 7


"James " <jfaghm@googlemail.com> wrote in message
news:i14stm$k24$1@fred.mathworks.com...
> Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> shifts. The result is 13 x 10512^2 matrix which it seems kills my
> computer.

A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
contiguous block of memory of size approximately 10.7 GB. Since you can't
work with that large a contiguous block of memory on a 32-bit system I'm
assuming your machine is a 64-bit system. Even on such a system, allocating
that much memory is going to take TIME.

> I'm using a MAC with 32GB of RAM and it's still crashing!

If it's crashing reproducibly, please send the crash report (and whatever
commands/data are necessary to reproduce the crash) to Technical Support.

If it's throwing an error indicating that MATLAB is out of memory, that's
different.

--
Steve Lord
slord@mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: Godzilla

Date: 9 Jul, 2010 03:24:03

Message: 3 of 7

"Steven Lord" <slord@mathworks.com> wrote in message <i153t5$qnf$1@fred.mathworks.com>...
>
> "James " <jfaghm@googlemail.com> wrote in message
> news:i14stm$k24$1@fred.mathworks.com...
> > Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> > shifts. The result is 13 x 10512^2 matrix which it seems kills my
> > computer.
>
> A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
> contiguous block of memory of size approximately 10.7 GB. Since you can't
> work with that large a contiguous block of memory on a 32-bit system I'm
> assuming your machine is a 64-bit system. Even on such a system, allocating
> that much memory is going to take TIME.
>
> > I'm using a MAC with 32GB of RAM and it's still crashing!
>
> If it's crashing reproducibly, please send the crash report (and whatever
> commands/data are necessary to reproduce the crash) to Technical Support.
>
> If it's throwing an error indicating that MATLAB is out of memory, that's
> different.
>
> --
> Steve Lord
> slord@mathworks.com
> comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> To contact Technical Support use the Contact Us link on
> http://www.mathworks.com
>

have you considered low-pass filtering the data and then decimation?

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: James

Date: 9 Jul, 2010 16:00:40

Message: 4 of 7

Steven,
Thank you for the response. There is no crash per se. After a while, Mac OS just opens a prompt telling me I'm running out of memory and must "Force Quit" MATLAB. Also sometimes, I get the Outofmemory error for time to time.
James


"Steven Lord" <slord@mathworks.com> wrote in message <i153t5$qnf$1@fred.mathworks.com>...
>
> "James " <jfaghm@googlemail.com> wrote in message
> news:i14stm$k24$1@fred.mathworks.com...
> > Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> > shifts. The result is 13 x 10512^2 matrix which it seems kills my
> > computer.
>
> A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
> contiguous block of memory of size approximately 10.7 GB. Since you can't
> work with that large a contiguous block of memory on a 32-bit system I'm
> assuming your machine is a 64-bit system. Even on such a system, allocating
> that much memory is going to take TIME.
>
> > I'm using a MAC with 32GB of RAM and it's still crashing!
>
> If it's crashing reproducibly, please send the crash report (and whatever
> commands/data are necessary to reproduce the crash) to Technical Support.
>
> If it's throwing an error indicating that MATLAB is out of memory, that's
> different.
>
> --
> Steve Lord
> slord@mathworks.com
> comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> To contact Technical Support use the Contact Us link on
> http://www.mathworks.com
>

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: James

Date: 9 Jul, 2010 16:05:04

Message: 5 of 7

Thank you Godzilla for your response. I'm still getting acquainted with time-series.
Are you referring to the y = decimate(x,r) method?
Thanks.
James

"Godzilla " <godzilla@tokyo.edu> wrote in message <i164oj$og8$1@fred.mathworks.com>...
> "Steven Lord" <slord@mathworks.com> wrote in message <i153t5$qnf$1@fred.mathworks.com>...
> >
> > "James " <jfaghm@googlemail.com> wrote in message
> > news:i14stm$k24$1@fred.mathworks.com...
> > > Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> > > shifts. The result is 13 x 10512^2 matrix which it seems kills my
> > > computer.
> >
> > A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
> > contiguous block of memory of size approximately 10.7 GB. Since you can't
> > work with that large a contiguous block of memory on a 32-bit system I'm
> > assuming your machine is a 64-bit system. Even on such a system, allocating
> > that much memory is going to take TIME.
> >
> > > I'm using a MAC with 32GB of RAM and it's still crashing!
> >
> > If it's crashing reproducibly, please send the crash report (and whatever
> > commands/data are necessary to reproduce the crash) to Technical Support.
> >
> > If it's throwing an error indicating that MATLAB is out of memory, that's
> > different.
> >
> > --
> > Steve Lord
> > slord@mathworks.com
> > comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> > To contact Technical Support use the Contact Us link on
> > http://www.mathworks.com
> >
>
> have you considered low-pass filtering the data and then decimation?

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: Wayne King

Date: 9 Jul, 2010 17:01:04

Message: 6 of 7

"James " <jfaghm@googlemail.com> wrote in message <i17hbg$ktl$1@fred.mathworks.com>...
> Thank you Godzilla for your response. I'm still getting acquainted with time-series.
> Are you referring to the y = decimate(x,r) method?
> Thanks.
> James
>
> "Godzilla " <godzilla@tokyo.edu> wrote in message <i164oj$og8$1@fred.mathworks.com>...
> > "Steven Lord" <slord@mathworks.com> wrote in message <i153t5$qnf$1@fred.mathworks.com>...
> > >
> > > "James " <jfaghm@googlemail.com> wrote in message
> > > news:i14stm$k24$1@fred.mathworks.com...
> > > > Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> > > > shifts. The result is 13 x 10512^2 matrix which it seems kills my
> > > > computer.
> > >
> > > A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
> > > contiguous block of memory of size approximately 10.7 GB. Since you can't
> > > work with that large a contiguous block of memory on a 32-bit system I'm
> > > assuming your machine is a 64-bit system. Even on such a system, allocating
> > > that much memory is going to take TIME.
> > >
> > > > I'm using a MAC with 32GB of RAM and it's still crashing!
> > >
> > > If it's crashing reproducibly, please send the crash report (and whatever
> > > commands/data are necessary to reproduce the crash) to Technical Support.
> > >
> > > If it's throwing an error indicating that MATLAB is out of memory, that's
> > > different.
> > >
> > > --
> > > Steve Lord
> > > slord@mathworks.com
> > > comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> > > To contact Technical Support use the Contact Us link on
> > > http://www.mathworks.com
> > >
> >
> > have you considered low-pass filtering the data and then decimation?

Hi James, yes decimate low pass filters the input data and then downsamples. Just be aware that lowpass filtering smooths your data, and the higher the decimation factor, the lower the cutoff frequency for the lowpass filter must be to prevent aliasing.

Wayne



Wayne

Subject: Using xcorr on a 10512 x 10512 with 6 lags

From: James

Date: 16 Jul, 2010 23:10:21

Message: 7 of 7

Hi Wayne,
Thank you for your reply. Actually, the issue with my data isn't with the frequency but w/ the number of data points. Each time series is only 744 points long, but I have 10512 time series and I want to compute the cross-correlation with 6 month lags.
So when I do a a cross-correlation of two time series everything is fine, it's when I try to use all of them that the computer just dies.
James
"Wayne King" <wmkingty@gmail.com> wrote in message <i17kkg$aaq$1@fred.mathworks.com>...
> "James " <jfaghm@googlemail.com> wrote in message <i17hbg$ktl$1@fred.mathworks.com>...
> > Thank you Godzilla for your response. I'm still getting acquainted with time-series.
> > Are you referring to the y = decimate(x,r) method?
> > Thanks.
> > James
> >
> > "Godzilla " <godzilla@tokyo.edu> wrote in message <i164oj$og8$1@fred.mathworks.com>...
> > > "Steven Lord" <slord@mathworks.com> wrote in message <i153t5$qnf$1@fred.mathworks.com>...
> > > >
> > > > "James " <jfaghm@googlemail.com> wrote in message
> > > > news:i14stm$k24$1@fred.mathworks.com...
> > > > > Hi I'm trying to get auto-correlation of a 10512 x 120 matrix with 6
> > > > > shifts. The result is 13 x 10512^2 matrix which it seems kills my
> > > > > computer.
> > > >
> > > > A 13-by-110502144 (= 10512^2) real full double precision matrix requires a
> > > > contiguous block of memory of size approximately 10.7 GB. Since you can't
> > > > work with that large a contiguous block of memory on a 32-bit system I'm
> > > > assuming your machine is a 64-bit system. Even on such a system, allocating
> > > > that much memory is going to take TIME.
> > > >
> > > > > I'm using a MAC with 32GB of RAM and it's still crashing!
> > > >
> > > > If it's crashing reproducibly, please send the crash report (and whatever
> > > > commands/data are necessary to reproduce the crash) to Technical Support.
> > > >
> > > > If it's throwing an error indicating that MATLAB is out of memory, that's
> > > > different.
> > > >
> > > > --
> > > > Steve Lord
> > > > slord@mathworks.com
> > > > comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> > > > To contact Technical Support use the Contact Us link on
> > > > http://www.mathworks.com
> > > >
> > >
> > > have you considered low-pass filtering the data and then decimation?
>
> Hi James, yes decimate low pass filters the input data and then downsamples. Just be aware that lowpass filtering smooths your data, and the higher the decimation factor, the lower the cutoff frequency for the lowpass filter must be to prevent aliasing.
>
> Wayne
>
>
>
> Wayne

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us