MATLAB Central Newsreader - taking a long time

taking a long time

Travis — Sat, 29 Nov 2008 16:35:02 -0500

I am using the distance function to find the distance between two sets of lat/longs. The data sets are 700,700 and 309 points long. It has been running for over 10 hours. Is this normal for it to take this long? Is there a quicker way to do this?

Re: taking a long time

Travis — Sun, 30 Nov 2008 00:44:02 -0500

OK, going on 24+ hours now. Again, is this normal? Is there a faster way?

Re: taking a long time

Roger Stafford — Sun, 30 Nov 2008 02:00:18 -0500

"Travis" <sinusoid2@hotmail.com> wrote in message <ggrqvm$k5l$1@fred.mathworks.com>...
> I am using the distance function to find the distance between two sets of lat/longs. The data sets are 700,700 and 309 points long. It has been running for over 10 hours. Is this normal for it to take this long? Is there a quicker way to do this?
--------
> OK, going on 24+ hours now. Again, is this normal? Is there a faster way?
--------
  Travis, you are placing a heavy burden on your computer's computational capacity. Every distance between a 700,700-element set and a 309-element set amounts to a total of 216,516,300 distances altogether, and each one of these presumably involves a great circle arc distance between pairs of longitude and latitude quantities. These are spherical trigonometric computations rather than the much simpler Euclidean formulas. If you are using an ellipsoidal earth, these computations would be even more time-consuming.

  I would suggest you time your computer for computing, say, all the distances between two 100-element sets and then multiply that by a factor of twenty-one thousand or thereabouts to get the timing you might reasonably expect for your problem. If the 100 by a 100 case takes only five seconds, you could expect to have six hours more to go beyond your present 24 hour wait.

  As for there being a faster method, some of the trigonometric calculations, such as determining the sine and cosine of the longitude and latitude quantities could be pre-computed just once for each set's elements and this would save some time as compared with starting each distance computation from scratch. However, I would expect that the people at Mathworks have already figured that out for themselves and would be taking such a shortcut, provided it is being done in a single call on the appropriate distance function.

  There is of course another (unlikely?) possibility. Your computer may not possess enough RAM memory to store the necessary large number of distances and it is doing a switch to virtual disk memory. That would certainly slow things down. Mathworks people can tell you whether such a thing might be happening. I always thought that Matlab would complain if it didn't possess a sufficient amount of contiguous RAM memory to store an allocated array.

Roger Stafford

Re: taking a long time

Travis — Sun, 30 Nov 2008 03:20:03 -0500

"Roger Stafford" <ellieandrogerxyzzy@mindspring.com.invalid> wrote in message <ggss3i$9vo$1@fred.mathworks.com>...
> "Travis" <sinusoid2@hotmail.com> wrote in message <ggrqvm$k5l$1@fred.mathworks.com>...
> > I am using the distance function to find the distance between two sets of lat/longs. The data sets are 700,700 and 309 points long. It has been running for over 10 hours. Is this normal for it to take this long? Is there a quicker way to do this?
> --------
> > OK, going on 24+ hours now. Again, is this normal? Is there a faster way?
> --------
> Travis, you are placing a heavy burden on your computer's computational capacity. Every distance between a 700,700-element set and a 309-element set amounts to a total of 216,516,300 distances altogether, and each one of these presumably involves a great circle arc distance between pairs of longitude and latitude quantities. These are spherical trigonometric computations rather than the much simpler Euclidean formulas. If you are using an ellipsoidal earth, these computations would be even more time-consuming.
>
> I would suggest you time your computer for computing, say, all the distances between two 100-element sets and then multiply that by a factor of twenty-one thousand or thereabouts to get the timing you might reasonably expect for your problem. If the 100 by a 100 case takes only five seconds, you could expect to have six hours more to go beyond your present 24 hour wait.
>
> As for there being a faster method, some of the trigonometric calculations, such as determining the sine and cosine of the longitude and latitude quantities could be pre-computed just once for each set's elements and this would save some time as compared with starting each distance computation from scratch. However, I would expect that the people at Mathworks have already figured that out for themselves and would be taking such a shortcut, provided it is being done in a single call on the appropriate distance function.
>
> There is of course another (unlikely?) possibility. Your computer may not possess enough RAM memory to store the necessary large number of distances and it is doing a switch to virtual disk memory. That would certainly slow things down. Mathworks people can tell you whether such a thing might be happening. I always thought that Matlab would complain if it didn't possess a sufficient amount of contiguous RAM memory to store an allocated array.
>
> Roger Stafford

Well, I knew it was a huge amount of number, but I was hoping with an AMD 6400 OC'd and 8 gb of good RAM it might not take so long. I think on my next run I will cut it down a little. It doesn't seem to be running out of memory, I have seen it use as much as 1.5gb at once though.