http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393
MATLAB Central Newsreader  Index matrix to sparse matrix... saving memory
Feed for thread: Index matrix to sparse matrix... saving memory
enus
©19942015 by MathWorks, Inc.
webmaster@mathworks.com
MATLAB Central Newsreader
http://blogs.law.harvard.edu/tech/rss
60
MathWorks
http://www.mathworks.com/images/membrane_icon.gif

Sat, 29 May 2010 16:30:23 +0000
Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750020
Anthony Hopf
Latest problem....<br>
<br>
I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.<br>
<br>
S = [1 2 3 0 0 0 0 0 0<br>
4 5 6 0 0 0 0 0 0<br>
7 8 9 10 11 12 13 14 0<br>
15 16 17 18 19 20 21 22 23]<br>
<br>
I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!<br>
<br>
S_s = sparse(S);%< Much less memory (in the above case there isn't much savings)<br>
<br>
Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?<br>
<br>
I am trying to figure out if I can use Bruno Luong's Sparse Sub Access, but I don't have su access to the server I am using to test it out...<br>
<br>
Thanks in advance

Sat, 29 May 2010 16:43:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750025
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...<br>
> Latest problem....<br>
> <br>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.<br>
> <br>
> S = [1 2 3 0 0 0 0 0 0<br>
> 4 5 6 0 0 0 0 0 0<br>
> 7 8 9 10 11 12 13 14 0<br>
> 15 16 17 18 19 20 21 22 23]<br>
> <br>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!<br>
> <br>
> S_s = sparse(S);%< Much less memory (in the above case there isn't much savings)<br>
> <br>
> Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?<br>
<br>
The following command will return the nonzeros elements of matrix: <br>
<br>
nonzeros(S_s(row,:))<br>
<br>
I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.<br>
<br>
AFAICS, you don not need my sparse access tool.<br>
<br>
Bruno

Sat, 29 May 2010 16:58:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750027
Anthony Hopf
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <htrg6p$2h3$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...<br>
> > Latest problem....<br>
> > <br>
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.<br>
> > <br>
> > S = [1 2 3 0 0 0 0 0 0<br>
> > 4 5 6 0 0 0 0 0 0<br>
> > 7 8 9 10 11 12 13 14 0<br>
> > 15 16 17 18 19 20 21 22 23]<br>
> > <br>
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!<br>
> > <br>
> > S_s = sparse(S);%< Much less memory (in the above case there isn't much savings)<br>
> > <br>
> > Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?<br>
> <br>
> The following command will return the nonzeros elements of matrix: <br>
> <br>
> nonzeros(S_s(row,:))<br>
> <br>
> I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.<br>
> <br>
> AFAICS, you don not need my sparse access tool.<br>
> <br>
> Bruno<br>
<br>
Bruno,<br>
<br>
That is a solid!! And you are right, transpose and it screams.<br>
<br>
Thank you again!!<br>
<br>
Anthony

Sat, 29 May 2010 17:49:22 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750037
Matt J
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...<br>
> Latest problem....<br>
> <br>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.<br>
> <br>
> S = [1 2 3 0 0 0 0 0 0<br>
> 4 5 6 0 0 0 0 0 0<br>
> 7 8 9 10 11 12 13 14 0<br>
> 15 16 17 18 19 20 21 22 23]<br>
> <br>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!<br>
==================<br>
<br>
Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?<br>
<br>
That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.

Sat, 29 May 2010 18:10:21 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750046
Anthony Hopf
"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrk32$2vm$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...<br>
> > Latest problem....<br>
> > <br>
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.<br>
> > <br>
> > S = [1 2 3 0 0 0 0 0 0<br>
> > 4 5 6 0 0 0 0 0 0<br>
> > 7 8 9 10 11 12 13 14 0<br>
> > 15 16 17 18 19 20 21 22 23]<br>
> > <br>
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!<br>
> ==================<br>
> <br>
> Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?<br>
> <br>
> That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.<br>
<br>
Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).<br>
<br>
I'll think about it, maybe I can apply this technique... Thanks again!<br>
<br>
Anthony

Sat, 29 May 2010 18:28:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750051
Matt J
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...<br>
<br>
> Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).<br>
> <br>
> I'll think about it, maybe I can apply this technique... Thanks again!<br>
======<br>
<br>
Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.<br>
<br>

Sat, 29 May 2010 21:56:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750088
Anthony Hopf
"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrmbl$nn6$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...<br>
> <br>
> > Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).<br>
> > <br>
> > I'll think about it, maybe I can apply this technique... Thanks again!<br>
> ======<br>
> <br>
> Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.<br>
> <br>
> <br>
<br>
I have two questions concerning the above and something else that goes to the above<br>
<br>
1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:<br>
<br>
for v=1:length(r_bin)<br>
index_temp = find( r_3d >= rL(v) & r_3d <rH(v));<br>
index_range(v,1:length(index_temp)) = index_temp;<br>
clear index_temp;<br>
end<br>
<br>
The reason I am asking is because I don't really know how many points there will be in each vector index_temp... so I end up not being able to allocate all the space I would like in index_range... can I do this with a sparse matrix, or should I just do what I am doing here and make index_range a sparse matrix afterwords? (rL and rH are the upper and lower bounds of the range bin)<br>
<br>
2. Given what you see here in number 1 I don't understand how I could create a logical array to do the same thing since each vector indexes points to different parts within r_3d and other matrices I'm not showing... Matt could you give me a clue?<br>
<br>
Thanks again<br>
<br>
Anthony

Sat, 29 May 2010 23:48:08 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750105
Matt J
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...<br>
> "Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrmbl$nn6$1@fred.mathworks.com>...<br>
> > "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...<br>
> > <br>
> > > Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).<br>
> > > <br>
> > > I'll think about it, maybe I can apply this technique... Thanks again!<br>
> > ======<br>
> > <br>
> > Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.<br>
> > <br>
> > <br>
> <br>
> I have two questions concerning the above and something else that goes to the above<br>
> <br>
> 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:<br>
> <br>
> for v=1:length(r_bin)<br>
> index_temp = find( r_3d >= rL(v) & r_3d <rH(v));<br>
> index_range(v,1:length(index_temp)) = index_temp;<br>
> clear index_temp;<br>
> end<br>
> <br>
> The reason I am asking is because I don't really know how many points there will be in each vector index_temp... so I end up not being able to allocate all the space I would like in index_range... can I do this with a sparse matrix, or should I just do what I am doing here and make index_range a sparse matrix afterwords? (rL and rH are the upper and lower bounds of the range bin)<br>
> <br>
> 2. Given what you see here in number 1 I don't understand how I could create a logical array to do the same thing since each vector indexes points to different parts within r_3d and other matrices I'm not showing... Matt could you give me a clue?<br>
================<br>
<br>
Combining all the recommendations in this thread (e.g., loading the indices columnwise instead of rowwise), I get to the following:<br>
<br>
<br>
%preallocate<br>
N=length(r_bin);<br>
I=cell(1,N); <br>
J=I;<br>
<br>
<br>
%compute sparseindices<br>
for v=1:N<br>
<br>
I{v}=find( r_3d >= rL(v) & r_3d <rH(v) );<br>
<br>
J{v}=I{v}; <br>
J{v}(:)=v;<br>
<br>
end<br>
<br>
index_range=sparse([I{:}], [J{:}],true); %final sparse logical matrix

Sun, 30 May 2010 09:07:06 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750162
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...<br>
> <br>
> <br>
> 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:<br>
> <br>
> for v=1:length(r_bin)<br>
> index_temp = find( r_3d >= rL(v) & r_3d <rH(v));<br>
> index_range(v,1:length(index_temp)) = index_temp;<br>
> clear index_temp;<br>
> end<br>
> <br>
<br>
You should definitively learn how to use function HISTC and family.<br>
<br>
Bruno

Sun, 30 May 2010 11:25:10 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750191
Anthony Hopf
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <htt9rq$1s8$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...<br>
> > <br>
> > <br>
> > 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:<br>
> > <br>
> > for v=1:length(r_bin)<br>
> > index_temp = find( r_3d >= rL(v) & r_3d <rH(v));<br>
> > index_range(v,1:length(index_temp)) = index_temp;<br>
> > clear index_temp;<br>
> > end<br>
> > <br>
> <br>
> You should definitively learn how to use function HISTC and family.<br>
> <br>
> Bruno<br>
<br>
Bruno,<br>
<br>
I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?<br>
<br>
In the code above I am doing the first step of the indexing, indexing in range. r_3d is the range of (r,theta,phi) to each point within a 3d cartesian space referenced to some point within the 3d space. Can you work with me on how I would go about utilizing these functions to solve such a problem?<br>
<br>
Thank you for the continued help, You guys are truly a essential piece to the matlab community!<br>
<br>
Anthony

Sun, 30 May 2010 12:02:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750195
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...<br>
<br>
> <br>
> Bruno,<br>
> <br>
> I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?<br>
<br>
If problem address in this thread needs can be solved with the builtin function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.<br>
<br>
The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple forloops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.<br>
<br>
Bruno

Sun, 30 May 2010 12:20:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#750198
Anthony Hopf
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <httk3s$hb0$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...<br>
> <br>
> > <br>
> > Bruno,<br>
> > <br>
> > I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?<br>
> <br>
> If problem address in this thread needs can be solved with the builtin function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.<br>
> <br>
> The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple forloops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.<br>
> <br>
> Bruno<br>
<br>
Bruno,<br>
<br>
I think you are right on with your statement "It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way." I will look back at the code you had posted before and think about this problem in a more abstract way.<br>
<br>
Thanks again,<br>
<br>
Anthony

Tue, 08 Jun 2010 19:47:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#752731
Anthony Hopf
Sorry to "double" post this (seems to fit here much better than the other thread)... but I think I am starting to see the light on this today... I haven't looked at it in awhile. for the case of binning in range, code shown above, I am using this:<br>
<br>
r_bin = (bin_r/2:bin_r:bin_r_max*bin_r+bin_r/2);%[m]<br>
[n,bin] = histc(r_3d(:),r_bin);<br>
<br>
This gives me n and bin with all the information I need, I am playing with it to figure out how to extract the index values so I have a n x max(n) matrix. But can't I use your histcn function to do all the binning I need to do... r,theta, and phi... at once like this?<br>
<br>
r_bin = (bin_r/2:bin_r:bin_r_max*bin_r+bin_r/2);%[m]<br>
theta_bin = (bin_t/2:bin_t:bin_t_max*bin_size+bin_t/2);%[deg]<br>
phi_bin = (bin_p/2:bin_p:bin_p_max*bin_p+bin_p/2);%[deg]<br>
<br>
hugearray = [r_3d(:) theta_3d(:) phi_3d(:) ];<br>
[count edges mid loc] = histcn(hugearray,r_bin,theta_bin,phi_bin);<br>
<br>
where count and loc give me essentially the same thing as n and bin that I can then try and place into a sparse matrix to store for later use?<br>
<br>
Thank you,<br>
<br>
Anthony<br>
<br>
<br>
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <httl5l$ns3$1@fred.mathworks.com>...<br>
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <httk3s$hb0$1@fred.mathworks.com>...<br>
> > "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...<br>
> > <br>
> > > <br>
> > > Bruno,<br>
> > > <br>
> > > I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?<br>
> > <br>
> > If problem address in this thread needs can be solved with the builtin function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.<br>
> > <br>
> > The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple forloops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.<br>
> > <br>
> > Bruno<br>
> <br>
> Bruno,<br>
> <br>
> I think you are right on with your statement "It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way." I will look back at the code you had posted before and think about this problem in a more abstract way.<br>
> <br>
> Thanks again,<br>
> <br>
> Anthony

Tue, 08 Jun 2010 20:25:20 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#752754
Bruno Luong
First, this a an monthold thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the missuse of sparse), the a lot time is waste.<br>
<br>
Now it will be the last time to write something on this topic, then I'll move on. Here we go:<br>
<br>
First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.<br>
<br>
%%<br>
function [ index_range ] = range_bin_sparse( r_4d, bin_size )<br>
%RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size<br>
% each radar is done individually and the ouput file is a sparse<br>
% matrix<br>
% [ range_index ] = range_bin_sparse( r_4d, bin_size )<br>
<br>
%single range bin binning of a single radar<br>
r_bin_max = ceil((((max(r_4d(:)))))/bin_size);<br>
r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]<br>
index_range = zeros(length(r_bin),250);<br>
<br>
%filter variables<br>
rH = r_bin + bin_size/2;<br>
rL = r_bin  bin_size/2;<br>
%find exceptable values for each range bin<br>
for v=1:length(r_bin)<br>
index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here<br>
index_range(v,1:length(index_temp)) = index_temp;<br>
end<br>
index_range = sparse(index_range');<br>
<br>
end<br>
<br>
% Now here is the data I use to test<br>
r_4d=peaks(50)+7; % + 7 to make data positive<br>
bin_size = 0.1;<br>
[ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
<br>
% Here is a way to use HISTC<br>
nbins = ceil(max(r_4d(:))/bin_size);<br>
edges = (0:nbins)*bin_size;<br>
[~,bin] = histc(r_4d,edges);<br>
bool = sparse(1:numel(bin), bin,true);<br>
<br>
% Note the the logical sparse matrix BOOL is not exactly like range, but it contains the<br>
% same information, if you want to find which index on the 50th bins, you can do this:<br>
<br>
>> index_range(:,50)<br>
<br>
ans =<br>
<br>
(1,1) 525<br>
(2,1) 526<br>
(3,1) 527<br>
(4,1) 528<br>
(5,1) 580<br>
(6,1) 826<br>
(7,1) 830<br>
(8,1) 1013<br>
(9,1) 1064<br>
(10,1) 1206<br>
(11,1) 1406<br>
(12,1) 1517<br>
<br>
% Or this:<br>
<br>
>> find(bool(:,50))<br>
<br>
ans =<br>
<br>
525<br>
526<br>
527<br>
528<br>
580<br>
826<br>
830<br>
1013<br>
1064<br>
1206<br>
1406<br>
1517<br>
<br>
% You see that they contain the same thing.<br>
<br>
In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins<br>
<br>
>> r_4d(bool(:,50)) <br>
<br>
ans =<br>
<br>
4.9948<br>
4.9059<br>
4.9003<br>
4.9737<br>
4.9775<br>
4.9397<br>
4.9658<br>
4.9965<br>
4.9447<br>
4.9068<br>
4.9602<br>
4.9365<br>
<br>
% Check if they fall inside the bins<br>
<br>
>> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)<br>
<br>
ans =<br>
<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
1<br>
<br>
>><br>
<br>
% Bruno

Tue, 08 Jun 2010 20:40:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#752758
Anthony Hopf
Thank you for your help, I apologize for my slow learning... you guys really are very good! <br>
<br>
Anthony<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...<br>
> First, this a an monthold thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the missuse of sparse), the a lot time is waste.<br>
> <br>
> Now it will be the last time to write something on this topic, then I'll move on. Here we go:<br>
> <br>
> First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.<br>
> <br>
> %%<br>
> function [ index_range ] = range_bin_sparse( r_4d, bin_size )<br>
> %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size<br>
> % each radar is done individually and the ouput file is a sparse<br>
> % matrix<br>
> % [ range_index ] = range_bin_sparse( r_4d, bin_size )<br>
> <br>
> %single range bin binning of a single radar<br>
> r_bin_max = ceil((((max(r_4d(:)))))/bin_size);<br>
> r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]<br>
> index_range = zeros(length(r_bin),250);<br>
> <br>
> %filter variables<br>
> rH = r_bin + bin_size/2;<br>
> rL = r_bin  bin_size/2;<br>
> %find exceptable values for each range bin<br>
> for v=1:length(r_bin)<br>
> index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here<br>
> index_range(v,1:length(index_temp)) = index_temp;<br>
> end<br>
> index_range = sparse(index_range');<br>
> <br>
> end<br>
> <br>
> % Now here is the data I use to test<br>
> r_4d=peaks(50)+7; % + 7 to make data positive<br>
> bin_size = 0.1;<br>
> [ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
> <br>
> % Here is a way to use HISTC<br>
> nbins = ceil(max(r_4d(:))/bin_size);<br>
> edges = (0:nbins)*bin_size;<br>
> [~,bin] = histc(r_4d,edges);<br>
> bool = sparse(1:numel(bin), bin,true);<br>
> <br>
> % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the<br>
> % same information, if you want to find which index on the 50th bins, you can do this:<br>
> <br>
> >> index_range(:,50)<br>
> <br>
> ans =<br>
> <br>
> (1,1) 525<br>
> (2,1) 526<br>
> (3,1) 527<br>
> (4,1) 528<br>
> (5,1) 580<br>
> (6,1) 826<br>
> (7,1) 830<br>
> (8,1) 1013<br>
> (9,1) 1064<br>
> (10,1) 1206<br>
> (11,1) 1406<br>
> (12,1) 1517<br>
> <br>
> % Or this:<br>
> <br>
> >> find(bool(:,50))<br>
> <br>
> ans =<br>
> <br>
> 525<br>
> 526<br>
> 527<br>
> 528<br>
> 580<br>
> 826<br>
> 830<br>
> 1013<br>
> 1064<br>
> 1206<br>
> 1406<br>
> 1517<br>
> <br>
> % You see that they contain the same thing.<br>
> <br>
> In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins<br>
> <br>
> >> r_4d(bool(:,50)) <br>
> <br>
> ans =<br>
> <br>
> 4.9948<br>
> 4.9059<br>
> 4.9003<br>
> 4.9737<br>
> 4.9775<br>
> 4.9397<br>
> 4.9658<br>
> 4.9965<br>
> 4.9447<br>
> 4.9068<br>
> 4.9602<br>
> 4.9365<br>
> <br>
> % Check if they fall inside the bins<br>
> <br>
> >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)<br>
> <br>
> ans =<br>
> <br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> <br>
> >><br>
> <br>
> % Bruno

Wed, 09 Jun 2010 14:21:07 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#752973
Anthony Hopf
Thank you Matt and Bruno!! I apologize for not staying with this topic, but I have quite a bit on my plate... and I am taking your advice, and TRYING to learn. You have both been very helpful and I hope you won't give up on me yet!!<br>
<br>
The reason I didn't stick with Matt's suggestion is I could never get values into the cells I and J, and then I got pulled from my work (circumstances that I do not wish to bring up).<br>
<br>
I am working with your suggestions and trying to apply the code to a 3d matrix, but first I started playing with your code to see what happens if there are NaN values within r and I run into a problem that I thought I could explain, the error I receive is:<br>
<br>
??? Error using ==> sparse<br>
Index into matrix must be positive.<br>
<br>
I realize the problem is that bin will have a 0 values due to the fact there is an out of bounds points... so I thought a remedy would be:<br>
<br>
r_4d=peaks(50)+7; % + 7 to make data positive<br>
r_4d(25:30,25:30) = NaN;<br>
bin_size = 0.1;<br>
%[ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
<br>
% Here is a way to use HISTC<br>
nbins = 151;%ceil(max(r_4d(:))/bin_size);<br>
r_bin = (0:nbins)*bin_size;<br>
[null,bin] = histc(r_4d,r_bin);<br>
bool = sparse(1:nnz(bin), nonzeros(bin),true);<br>
<br>
As you notice I have modified r_4d(25,25) to be a NaN value and commented out index_range (this line also perplexes me). My final two changes are in the sparse command where I try and compensate for the fact that there are out of bounds points... this seems to mess things up, when I go to a much larger data set with a 3d matrix r_4d the code runs, the number of nnz points in bool matches the number of ~isnan(r_4d) so I know histc is still working... but when I index using bool the returned values are not correct.<br>
<br>
for the above code:<br>
without NaNs: <br>
bool(:,50) <br>
<br>
ans = <br>
<br>
(525,1) 1 <br>
(526,1) 1<br>
(527,1) 1<br>
(528,1) 1<br>
(580,1) 1<br>
(826,1) 1<br>
(830,1) 1<br>
(1013,1) 1<br>
(1064,1) 1<br>
(1206,1) 1<br>
(1406,1) 1<br>
(1517,1) 1<br>
<br>
with NaNs:<br>
bool(:,50)<br>
<br>
ans =<br>
<br>
(525,1) 1<br>
(526,1) 1<br>
(527,1) 1<br>
(528,1) 1<br>
(580,1) 1<br>
(826,1) 1<br>
(830,1) 1<br>
(1013,1) 1<br>
(1064,1) 1<br>
(1206,1) 1<br>
(1382,1) 1<br>
(1481,1) 1<br>
<br>
Can you please give me a little more advice on this?<br>
<br>
Thank you<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...<br>
> First, this a an monthold thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the missuse of sparse), the a lot time is waste.<br>
> <br>
> Now it will be the last time to write something on this topic, then I'll move on. Here we go:<br>
> <br>
> First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.<br>
> <br>
> %%<br>
> function [ index_range ] = range_bin_sparse( r_4d, bin_size )<br>
> %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size<br>
> % each radar is done individually and the ouput file is a sparse<br>
> % matrix<br>
> % [ range_index ] = range_bin_sparse( r_4d, bin_size )<br>
> <br>
> %single range bin binning of a single radar<br>
> r_bin_max = ceil((((max(r_4d(:)))))/bin_size);<br>
> r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]<br>
> index_range = zeros(length(r_bin),250);<br>
> <br>
> %filter variables<br>
> rH = r_bin + bin_size/2;<br>
> rL = r_bin  bin_size/2;<br>
> %find exceptable values for each range bin<br>
> for v=1:length(r_bin)<br>
> index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here<br>
> index_range(v,1:length(index_temp)) = index_temp;<br>
> end<br>
> index_range = sparse(index_range');<br>
> <br>
> end<br>
> <br>
> % Now here is the data I use to test<br>
> r_4d=peaks(50)+7; % + 7 to make data positive<br>
> bin_size = 0.1;<br>
> [ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
> <br>
> % Here is a way to use HISTC<br>
> nbins = ceil(max(r_4d(:))/bin_size);<br>
> edges = (0:nbins)*bin_size;<br>
> [~,bin] = histc(r_4d,edges);<br>
> bool = sparse(1:numel(bin), bin,true);<br>
> <br>
> % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the<br>
> % same information, if you want to find which index on the 50th bins, you can do this:<br>
> <br>
> >> index_range(:,50)<br>
> <br>
> ans =<br>
> <br>
> (1,1) 525<br>
> (2,1) 526<br>
> (3,1) 527<br>
> (4,1) 528<br>
> (5,1) 580<br>
> (6,1) 826<br>
> (7,1) 830<br>
> (8,1) 1013<br>
> (9,1) 1064<br>
> (10,1) 1206<br>
> (11,1) 1406<br>
> (12,1) 1517<br>
> <br>
> % Or this:<br>
> <br>
> >> find(bool(:,50))<br>
> <br>
> ans =<br>
> <br>
> 525<br>
> 526<br>
> 527<br>
> 528<br>
> 580<br>
> 826<br>
> 830<br>
> 1013<br>
> 1064<br>
> 1206<br>
> 1406<br>
> 1517<br>
> <br>
> % You see that they contain the same thing.<br>
> <br>
> In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins<br>
> <br>
> >> r_4d(bool(:,50)) <br>
> <br>
> ans =<br>
> <br>
> 4.9948<br>
> 4.9059<br>
> 4.9003<br>
> 4.9737<br>
> 4.9775<br>
> 4.9397<br>
> 4.9658<br>
> 4.9965<br>
> 4.9447<br>
> 4.9068<br>
> 4.9602<br>
> 4.9365<br>
> <br>
> % Check if they fall inside the bins<br>
> <br>
> >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)<br>
> <br>
> ans =<br>
> <br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> 1<br>
> <br>
> >><br>
> <br>
> % Bruno

Wed, 09 Jun 2010 14:59:08 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#752988
Anthony Hopf
So I think a couple lines fixed the problem:<br>
<br>
%%%%%%<br>
bin_size = 100;<br>
r_bin_max = ceil((((nanmax(double(r_4d(:))))))/bin_size);<br>
r_bin = (0:bin_size:r_bin_max*bin_size);%[m]<br>
[null,bin] = histc(r_4d(:),r_bin);<br>
bin(bin==0)=max(bin(:))+1;<br>
bool = sparse(1:numel(bin), (bin),true);<br>
bool = bool(:,1:end1);<br>
<br>
I added the r_4d(:) so I can work with 3d matrices, and I took all the values of bin =0 and changed their value to something outside of the actual r_4d matrix index values and then lopped them off at the end.<br>
<br>
Again, Thank you for all of your help. Now to apply this to r,theta, and phi... I think i am going to do this sequentially.<br>
<br>
Anthony<br>
<br>
<br>
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <huo80j$c5n$1@fred.mathworks.com>...<br>
> Thank you Matt and Bruno!! I apologize for not staying with this topic, but I have quite a bit on my plate... and I am taking your advice, and TRYING to learn. You have both been very helpful and I hope you won't give up on me yet!!<br>
> <br>
> The reason I didn't stick with Matt's suggestion is I could never get values into the cells I and J, and then I got pulled from my work (circumstances that I do not wish to bring up).<br>
> <br>
> I am working with your suggestions and trying to apply the code to a 3d matrix, but first I started playing with your code to see what happens if there are NaN values within r and I run into a problem that I thought I could explain, the error I receive is:<br>
> <br>
> ??? Error using ==> sparse<br>
> Index into matrix must be positive.<br>
> <br>
> I realize the problem is that bin will have a 0 values due to the fact there is an out of bounds points... so I thought a remedy would be:<br>
> <br>
> r_4d=peaks(50)+7; % + 7 to make data positive<br>
> r_4d(25:30,25:30) = NaN;<br>
> bin_size = 0.1;<br>
> %[ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
> <br>
> % Here is a way to use HISTC<br>
> nbins = 151;%ceil(max(r_4d(:))/bin_size);<br>
> r_bin = (0:nbins)*bin_size;<br>
> [null,bin] = histc(r_4d,r_bin);<br>
> bool = sparse(1:nnz(bin), nonzeros(bin),true);<br>
> <br>
> As you notice I have modified r_4d(25,25) to be a NaN value and commented out index_range (this line also perplexes me). My final two changes are in the sparse command where I try and compensate for the fact that there are out of bounds points... this seems to mess things up, when I go to a much larger data set with a 3d matrix r_4d the code runs, the number of nnz points in bool matches the number of ~isnan(r_4d) so I know histc is still working... but when I index using bool the returned values are not correct.<br>
> <br>
> for the above code:<br>
> without NaNs: <br>
> bool(:,50) <br>
> <br>
> ans = <br>
> <br>
> (525,1) 1 <br>
> (526,1) 1<br>
> (527,1) 1<br>
> (528,1) 1<br>
> (580,1) 1<br>
> (826,1) 1<br>
> (830,1) 1<br>
> (1013,1) 1<br>
> (1064,1) 1<br>
> (1206,1) 1<br>
> (1406,1) 1<br>
> (1517,1) 1<br>
> <br>
> with NaNs:<br>
> bool(:,50)<br>
> <br>
> ans =<br>
> <br>
> (525,1) 1<br>
> (526,1) 1<br>
> (527,1) 1<br>
> (528,1) 1<br>
> (580,1) 1<br>
> (826,1) 1<br>
> (830,1) 1<br>
> (1013,1) 1<br>
> (1064,1) 1<br>
> (1206,1) 1<br>
> (1382,1) 1<br>
> (1481,1) 1<br>
> <br>
> Can you please give me a little more advice on this?<br>
> <br>
> Thank you<br>
> <br>
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...<br>
> > First, this a an monthold thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the missuse of sparse), the a lot time is waste.<br>
> > <br>
> > Now it will be the last time to write something on this topic, then I'll move on. Here we go:<br>
> > <br>
> > First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.<br>
> > <br>
> > %%<br>
> > function [ index_range ] = range_bin_sparse( r_4d, bin_size )<br>
> > %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size<br>
> > % each radar is done individually and the ouput file is a sparse<br>
> > % matrix<br>
> > % [ range_index ] = range_bin_sparse( r_4d, bin_size )<br>
> > <br>
> > %single range bin binning of a single radar<br>
> > r_bin_max = ceil((((max(r_4d(:)))))/bin_size);<br>
> > r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]<br>
> > index_range = zeros(length(r_bin),250);<br>
> > <br>
> > %filter variables<br>
> > rH = r_bin + bin_size/2;<br>
> > rL = r_bin  bin_size/2;<br>
> > %find exceptable values for each range bin<br>
> > for v=1:length(r_bin)<br>
> > index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here<br>
> > index_range(v,1:length(index_temp)) = index_temp;<br>
> > end<br>
> > index_range = sparse(index_range');<br>
> > <br>
> > end<br>
> > <br>
> > % Now here is the data I use to test<br>
> > r_4d=peaks(50)+7; % + 7 to make data positive<br>
> > bin_size = 0.1;<br>
> > [ index_range ] = range_bin_sparse( r_4d, bin_size );<br>
> > <br>
> > % Here is a way to use HISTC<br>
> > nbins = ceil(max(r_4d(:))/bin_size);<br>
> > edges = (0:nbins)*bin_size;<br>
> > [~,bin] = histc(r_4d,edges);<br>
> > bool = sparse(1:numel(bin), bin,true);<br>
> > <br>
> > % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the<br>
> > % same information, if you want to find which index on the 50th bins, you can do this:<br>
> > <br>
> > >> index_range(:,50)<br>
> > <br>
> > ans =<br>
> > <br>
> > (1,1) 525<br>
> > (2,1) 526<br>
> > (3,1) 527<br>
> > (4,1) 528<br>
> > (5,1) 580<br>
> > (6,1) 826<br>
> > (7,1) 830<br>
> > (8,1) 1013<br>
> > (9,1) 1064<br>
> > (10,1) 1206<br>
> > (11,1) 1406<br>
> > (12,1) 1517<br>
> > <br>
> > % Or this:<br>
> > <br>
> > >> find(bool(:,50))<br>
> > <br>
> > ans =<br>
> > <br>
> > 525<br>
> > 526<br>
> > 527<br>
> > 528<br>
> > 580<br>
> > 826<br>
> > 830<br>
> > 1013<br>
> > 1064<br>
> > 1206<br>
> > 1406<br>
> > 1517<br>
> > <br>
> > % You see that they contain the same thing.<br>
> > <br>
> > In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins<br>
> > <br>
> > >> r_4d(bool(:,50)) <br>
> > <br>
> > ans =<br>
> > <br>
> > 4.9948<br>
> > 4.9059<br>
> > 4.9003<br>
> > 4.9737<br>
> > 4.9775<br>
> > 4.9397<br>
> > 4.9658<br>
> > 4.9965<br>
> > 4.9447<br>
> > 4.9068<br>
> > 4.9602<br>
> > 4.9365<br>
> > <br>
> > % Check if they fall inside the bins<br>
> > <br>
> > >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)<br>
> > <br>
> > ans =<br>
> > <br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > 1<br>
> > <br>
> > >><br>
> > <br>
> > % Bruno

Wed, 09 Jun 2010 21:58:20 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753131
Anthony Hopf
I feel like I am sooo close....<br>
<br>
My issue is with finding the intersection between 3 sparse logical matrices<br>
<br>
idxr, idxa, and idxe are all sparse logical matrices. The r, a and e correspond to range, azimuth and elevation. With the help of Bruno and Matt I find them like this:<br>
<br>
r_bin_max = ceil((((nanmax(r_4d(:)))))/bin_size);<br>
r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]<br>
az_bin = 0:beamwidth:360;%[deg]<br>
el_bin = 0:beamwidth:thetae_max;%[deg]<br>
%<br>
[null,bin] = histc(r_4d(:),r_bin);<br>
clear null<br>
bin(bin==0)=max(bin(:))+1;<br>
idxr = sparse(1:numel(bin), (bin),true);<br>
idxr = idxr(:,1:end1);<br>
[null,ebin] = histc(el(:),el_bin);<br>
clear null<br>
ebin(ebin==0)=max(ebin(:))+1;<br>
idxe = sparse(1:numel(ebin), (ebin),true);<br>
idxe = idxe(:,1:end1);<br>
[null,abin] = histc(az(:),az_bin);<br>
clear null<br>
abin(abin==0)=max(abin(:))+1;<br>
idxa = sparse(1:numel(abin), (abin),true);<br>
idxa = idxa(:,1:end1);<br>
<br>
%% I was going to use histcn but this takes less than 0.5sec to complete with histc, the histc calls take ~0.05sec, a similar histcn call would be ~0.15sec.<br>
<br>
This is infinitely faster than the way I was doing it before, but I have the problem of finding the intersection between them to create the index representative of a 3d sampling in r,theta,phi... the way I do it now is:<br>
<br>
idxbr =( zeros(size(idxr,2)*size(idxa,2)*size(idxe,2),100));<br>
<br>
counter = 0;<br>
for iir = 1:size(idxr,2)<br>
for iia = 1:size(idxa,2)<br>
for iie = 1:size(idxe,2)<br>
idxrae = intersect(find(idxr(:,iir)),intersect(find(idxa(:,iia)),find(idxe(:,iie))));<br>
counter = counter+1;<br>
idxbr(1:length(idxrae),counter) = idxrae;<br>
clear idxrae<br>
end%***end of beam position second loop<br>
end%***end of range first loop<br>
end<br>
idxbr = sparse(idxbr);<br>
end%function end<br>
<br>
This not only takes an extremely long time for large idx* arrays, but I loss the excellent logical sparse arrays Bruno and Matt taught me how to make.<br>
<br>
if it is any consolation I know everything I need is in rbin, ebin, and abin... I can see it when I print the values out, the bins are right there to use.<br>
<br>
Can I get rid of these for loops?<br>
<br>
Thanks, Anthony

Thu, 10 Jun 2010 09:28:03 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753234
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hup2ps$rmb$1@fred.mathworks.com>...<br>
<br>
> <br>
> Can I get rid of these for loops?<br>
<br>
For multidimensional histogram use Fex submission HISTCN.<br>
<br>
Bruno

Thu, 10 Jun 2010 18:58:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753466
Anthony Hopf
Bruno or anyone that has looked through the thread,<br>
<br>
I'm beating myself up here... I just can't see through to the end. Histcn returns the same thing Histc does is return one output "loc" rather than multiple outputs of "bin." If I cat the bins <br>
<br>
[bin1 bin2 bin3] == [ loc ]<br>
<br>
Where I hit a wall is making use of loc. Loc has all of my bin locations, which I need and it does it very quickly, but I need to relate each column of "loc" to one another, compare every combination of bin value. This iterative comparison is what slows me down, and what I have been stuck on conceptually, which was what originally slowed down my understanding of your function histcn.<br>
<br>
Does anyone know of a function that will compress these for loop after I have used histcn to find the bin values? The for loops step through the bin values of each column of "loc" returning the correct index values of "loc".<br>
<br>
%I previously used histcn to create "loc"<br>
idxbr =( zeros(length(azm2)*length(el2)*length(rb2),100));<br>
counter = 0;<br>
<br>
for iia = 1:max(loc(:,1)) %stepping through loc(:,1)<br>
for iie = 1:max(loc(:,2)) %stepping through loc(:,2)<br>
for iir = 1:max(loc(:,3)) %stepping through loc(:,3)<br>
counter = counter+1;<br>
tempfind = find(loc2(:,1)==iia & loc2(:,2)==iie & loc2(:,3) == iir); %find all values corresponding to loc = [1 1 1] to loc = [max(loc(:,1)) max(loc(:,2)) max(loc(:,3))] and everything in between.<br>
idxbr(1:length(tempfind),counter) = tempfind(:); %store the indexes<br>
clear tempfind<br>
end<br>
end<br>
end<br>
<br>
My buddy does this kind of lookup in SQL databases, but I don't know how to implement it in Matlab... I would assume there is a function, trick, or something I missed in histcn that someone can hint to? It I just had to find all points that sit in bin = 1, 2, 3, .... then this would be very simple, but because I have to look at each combination of bins it makes it tricky.<br>
<br>
I realize this is a long thread but I hope someone reads this and can give me a hint!!<br>
<br>
It is greatly appreciated.<br>
<br>
Anthony<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <huqb73$q0d$1@fred.mathworks.com>...<br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hup2ps$rmb$1@fred.mathworks.com>...<br>
> <br>
> > <br>
> > Can I get rid of these for loops?<br>
> <br>
> For multidimensional histogram use Fex submission HISTCN.<br>
> <br>
> Bruno

Thu, 10 Jun 2010 19:18:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753473
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurcjs$h31$1@fred.mathworks.com>...<br>
> Bruno or anyone that has looked through the thread,<br>
> <br>
> I'm beating myself up here... I just can't see through to the end. Histcn returns the same thing Histc does is return one output "loc" rather than multiple outputs of "bin." If I cat the bins <br>
> <br>
> [bin1 bin2 bin3] == [ loc ]<br>
> <br>
> Where I hit a wall is making use of loc. Loc has all of my bin locations, which I need and it does it very quickly, but I need to relate each column of "loc" to one another, compare every combination of bin value. This iterative comparison is what slows me down, and what I have been stuck on conceptually, which was what originally slowed down my understanding of your function histcn.<br>
> <br>
> Does anyone know of a function that will compress these for loop after I have used histcn to find the bin values? The for loops step through the bin values of each column of "loc" returning the correct index values of "loc".<br>
<br>
I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTC. In this thread, the ACCUMARRAY does carry out the sum (because that was originally asked), but you can adapt it to your specific need, including find all the indexes that belong to the ndimensional bins.<br>
<br>
Again, that was answered in Apr 27th, 2010.<br>
<br>
Bruno

Thu, 10 Jun 2010 19:38:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753483
Bruno Luong
I wrote:<br>
> <br>
> I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...<br>
<br>
Sorry, should read "HISTCN"<br>
<br>
Bruno

Thu, 10 Jun 2010 19:57:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#753493
Anthony Hopf
I understood, and appreciate your help. accumarray it is!!<br>
<br>
I'll post my solution when I figure this function out.<br>
<br>
Anthony<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...<br>
> I wrote:<br>
> > <br>
> > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...<br>
> <br>
> Sorry, should read "HISTCN"<br>
> <br>
> Bruno

Sun, 13 Jun 2010 14:45:06 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#754073
Anthony Hopf
Bruno... I can't believe I missed the funcitonality of histcn and accumarray!! This definitely gets me closer to my ultimate result.<br>
<br>
The below countermatrix returns the number of points within each bin given by histcn... This works extremely fast even for very large 3d matrices. I ran into a bit of a problem with "out of range" points, but dealt with these by tossing them in outer bins and taking them off of the countermatrix in the final steps. (code is at the bottom of this post).<br>
<br>
This matrix isn't exactly what I was looking for though... <br>
<br>
1. I'm curious if there is a way to collect the index values, corresponding to the original val3d matrix, into bin vectors. What I would like is a vector of index values for each point within countermatrix, at this point the structure could be arbitrary but optimally it could be indexed by each bin value... cell arrays of cell array, rbincell{tbincell{pbincell{[idx_vals for countermatrix(r,t,p)]}}} (This is my first time working with the ideas of cells). Keeping the index values accumulated seems doable with accumarray because it "touches" each one of these values... the array being operated on by accumarray would just be the index values corresponding to the points within val3d, rather than val3d itself<br>
<br>
2. Given the above I would also like to fill the original matrix val3d with the accumulated results, for example if val3d(i,j,k) fell into the sum corresponding to countermatrix(r,t,p), then we would put that sum in countermatrix(r,t,p) back into val3d(i,j,k), and all other points that where used for that particular sum. I do this iteratively now after finding the matrix mentioned in 1 above... but again this is a for loop that I want to get rid of.<br>
<br>
here is the code that creates countermatrix, in this case val3d is an array of ones and we should find that count and countermatrix are equal:<br>
<br>
%variables<br>
x = 1e3.*(1:.1:2);<br>
y = 1e3.*(0:.1:1);<br>
z = 1e3.*(0:.1:1);<br>
<br>
[x3d,y3d,z3d] = meshgrid(x,y,z);<br>
[t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);<br>
%vector to accumulate from<br>
val3d = ones(size(x3d));<br>
<br>
%maximum values of the bins<br>
maxt = max(t3d(:));<br>
maxp = max(p3d(:))(pi/180);<br>
maxr = max(r3d(:));<br>
%bins <br>
tbin = linspace(0,maxt,6);<br>
pbin = linspace(0,maxp,6);<br>
rbin = linspace(1e3,maxr,6);<br>
%histogram<br>
data = [r3d(:) t3d(:) p3d(:)];<br>
<br>
[count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);<br>
<br>
% temp = loc(:,1);temp(temp ==0)=max(temp)+1;loc(:,1) = temp;<br>
<br>
%place "out of range" values at the max bin for each rbin,tbin,pbin<br>
nir = zeros(1,3);<br>
for oor = 1:3<br>
temp = loc(:,oor);<br>
if any(temp==0) %if there are out of range values<br>
temp(temp ==0)=max(temp)+1;<br>
loc(:,oor) = temp;<br>
clear temp<br>
nir(oor)=1;<br>
end<br>
end<br>
<br>
countermatrix_accum = accumarray(loc,val3d(:));<br>
<br>
%remove "out of range" accumulated bins<br>
if nir(1)<br>
countermatrix_accum = countermatrix_accum(1:end1,:,:);<br>
end<br>
if nir(2)<br>
countermatrix_accum = countermatrix_accum(:,1:end1,:);<br>
end<br>
if nir(3)<br>
countermatrix_accum = countermatrix_accum(:,:,1:end1);<br>
end<br>
<br>
isequal(count,countermatrix_accum)<br>
<br>
<br>
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurg2g$5kt$1@fred.mathworks.com>...<br>
> I understood, and appreciate your help. accumarray it is!!<br>
> <br>
> I'll post my solution when I figure this function out.<br>
> <br>
> Anthony<br>
> <br>
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...<br>
> > I wrote:<br>
> > > <br>
> > > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...<br>
> > <br>
> > Sorry, should read "HISTCN"<br>
> > <br>
> > Bruno

Mon, 14 Jun 2010 01:17:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#754130
Anthony Hopf
Thank you very much Bruno, and others, for giving your advise... <br>
<br>
I had all of the resources to accomplish number 2... haven't figured out number 1 yet. Here is the code I am working with now, in case you are curious, i'm sure it isn't optimized:<br>
<br>
%variables<br>
x = 1e3.*(1:.1:2);<br>
y = 1e3.*(0:.1:1);<br>
z = 1e3.*(0:.1:1);<br>
<br>
[x3d,y3d,z3d] = meshgrid(x,y,z);<br>
[t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);<br>
%vector to accumulate from<br>
val3d = ones(size(x3d));<br>
<br>
%maximum values of the bins<br>
maxt = max(t3d(:));<br>
maxp = max(p3d(:))(pi/180);<br>
maxr = max(r3d(:));<br>
%bins <br>
tbin = linspace(0,maxt,6);<br>
pbin = linspace(0,maxp,6);<br>
rbin = linspace(1e3,maxr,6);<br>
%histogram<br>
data = [r3d(:) t3d(:) p3d(:)];<br>
<br>
[count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);<br>
<br>
%place "out of range" values at the max bin for each rbin,tbin,pbin<br>
nir = zeros(1,3);<br>
for oor = 1:3<br>
temp = loc(:,oor);<br>
if any(temp==0) %if there are out of range values<br>
temp(temp ==0)=max(temp)+1;<br>
loc(:,oor) = temp;<br>
clear temp<br>
nir(oor)=1;<br>
end<br>
end<br>
<br>
countermatrix_accum = accumarray(loc,val3d(:));<br>
<br>
%set countermatrix "out of range" values to zero<br>
if nir(1)<br>
countermatrix_accum(end,:,:) = 0;<br>
end<br>
if nir(2)<br>
countermatrix_accum(:,end,:) = 0;<br>
end<br>
if nir(3)<br>
countermatrix_accum(:,:,end) = 0;<br>
end<br>
<br>
%g will be the number of points at each common grid point<br>
gidx = sub2ind(size(countermatrix_accum),loc(:,1),loc(:,2),loc(:,3));<br>
<br>
g = countermatrix_accum(gidx);<br>
<br>
%reshape g to get our matrix<br>
newval3d = reshape(g,size(val3d));<br>
<br>
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hv2qti$j1l$1@fred.mathworks.com>...<br>
> Bruno... I can't believe I missed the funcitonality of histcn and accumarray!! This definitely gets me closer to my ultimate result.<br>
> <br>
> The below countermatrix returns the number of points within each bin given by histcn... This works extremely fast even for very large 3d matrices. I ran into a bit of a problem with "out of range" points, but dealt with these by tossing them in outer bins and taking them off of the countermatrix in the final steps. (code is at the bottom of this post).<br>
> <br>
> This matrix isn't exactly what I was looking for though... <br>
> <br>
> 1. I'm curious if there is a way to collect the index values, corresponding to the original val3d matrix, into bin vectors. What I would like is a vector of index values for each point within countermatrix, at this point the structure could be arbitrary but optimally it could be indexed by each bin value... cell arrays of cell array, rbincell{tbincell{pbincell{[idx_vals for countermatrix(r,t,p)]}}} (This is my first time working with the ideas of cells). Keeping the index values accumulated seems doable with accumarray because it "touches" each one of these values... the array being operated on by accumarray would just be the index values corresponding to the points within val3d, rather than val3d itself<br>
> <br>
> 2. Given the above I would also like to fill the original matrix val3d with the accumulated results, for example if val3d(i,j,k) fell into the sum corresponding to countermatrix(r,t,p), then we would put that sum in countermatrix(r,t,p) back into val3d(i,j,k), and all other points that where used for that particular sum. I do this iteratively now after finding the matrix mentioned in 1 above... but again this is a for loop that I want to get rid of.<br>
> <br>
> here is the code that creates countermatrix, in this case val3d is an array of ones and we should find that count and countermatrix are equal:<br>
> <br>
> %variables<br>
> x = 1e3.*(1:.1:2);<br>
> y = 1e3.*(0:.1:1);<br>
> z = 1e3.*(0:.1:1);<br>
> <br>
> [x3d,y3d,z3d] = meshgrid(x,y,z);<br>
> [t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);<br>
> %vector to accumulate from<br>
> val3d = ones(size(x3d));<br>
> <br>
> %maximum values of the bins<br>
> maxt = max(t3d(:));<br>
> maxp = max(p3d(:))(pi/180);<br>
> maxr = max(r3d(:));<br>
> %bins <br>
> tbin = linspace(0,maxt,6);<br>
> pbin = linspace(0,maxp,6);<br>
> rbin = linspace(1e3,maxr,6);<br>
> %histogram<br>
> data = [r3d(:) t3d(:) p3d(:)];<br>
> <br>
> [count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);<br>
> <br>
> % temp = loc(:,1);temp(temp ==0)=max(temp)+1;loc(:,1) = temp;<br>
> <br>
> %place "out of range" values at the max bin for each rbin,tbin,pbin<br>
> nir = zeros(1,3);<br>
> for oor = 1:3<br>
> temp = loc(:,oor);<br>
> if any(temp==0) %if there are out of range values<br>
> temp(temp ==0)=max(temp)+1;<br>
> loc(:,oor) = temp;<br>
> clear temp<br>
> nir(oor)=1;<br>
> end<br>
> end<br>
> <br>
> countermatrix_accum = accumarray(loc,val3d(:));<br>
> <br>
> %remove "out of range" accumulated bins<br>
> if nir(1)<br>
> countermatrix_accum = countermatrix_accum(1:end1,:,:);<br>
> end<br>
> if nir(2)<br>
> countermatrix_accum = countermatrix_accum(:,1:end1,:);<br>
> end<br>
> if nir(3)<br>
> countermatrix_accum = countermatrix_accum(:,:,1:end1);<br>
> end<br>
> <br>
> isequal(count,countermatrix_accum)<br>
> <br>
> <br>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurg2g$5kt$1@fred.mathworks.com>...<br>
> > I understood, and appreciate your help. accumarray it is!!<br>
> > <br>
> > I'll post my solution when I figure this function out.<br>
> > <br>
> > Anthony<br>
> > <br>
> > "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...<br>
> > > I wrote:<br>
> > > > <br>
> > > > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...<br>
> > > <br>
> > > Sorry, should read "HISTCN"<br>
> > > <br>
> > > Bruno

Mon, 14 Jun 2010 06:07:03 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#754155
Bruno Luong
Take a look at example 10 of the doc of accumarray:<br>
<a href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html">http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html</a><br>
<br>
In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.<br>
<br>
With something similar you can accomplish the task you want.<br>
<br>
Bruno

Mon, 14 Jun 2010 10:20:04 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#754187
Anthony Hopf
Thank you Bruno... The extreme speed up I see using accumarray may negate the need to bin all of the index values for storage. But I will play with this more.<br>
<br>
Anthony<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hv4gu7$ark$1@fred.mathworks.com>...<br>
> Take a look at example 10 of the doc of accumarray:<br>
> <a href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html">http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html</a><br>
> <br>
> In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.<br>
> <br>
> With something similar you can accomplish the task you want.<br>
> <br>
> Bruno

Thu, 15 Jul 2010 13:20:21 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#762831
Anthony Hopf
Bruno,<br>
<br>
Is there any reason that I would get different values for the output of histcn max(loc) and size(count)?<br>
<br>
I have not had to look back at this part of my code until I started messing with using the count output instead of doing my accumarray on a array of ones to calculate the number of points in each bin.<br>
<br>
for example<br>
I am getting max(loc) = [6 4 9] and size(count) = [7 4 9]<br>
<br>
I know there are "zero" values loc meaning some points where not binned, but that was always reflected in count too.<br>
<br>
Hope all is well,<br>
<br>
Anthony<br>
<br>
"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hv4gu7$ark$1@fred.mathworks.com>...<br>
> Take a look at example 10 of the doc of accumarray:<br>
> <a href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html">http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html</a><br>
> <br>
> In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.<br>
> <br>
> With something similar you can accomplish the task you want.<br>
> <br>
> Bruno

Fri, 16 Jul 2010 06:54:05 +0000
Re: Index matrix to sparse matrix... saving memory
http://www.mathworks.com/matlabcentral/newsreader/view_thread/283393#763080
Bruno Luong
"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <i1n1ul$mf0$1@fred.mathworks.com>...<br>
> Bruno,<br>
> <br>
> Is there any reason that I would get different values for the output of histcn max(loc) and size(count)?<br>
> <br>
> I have not had to look back at this part of my code until I started messing with using the count output instead of doing my accumarray on a array of ones to calculate the number of points in each bin.<br>
> <br>
> for example<br>
> I am getting max(loc) = [6 4 9] and size(count) = [7 4 9]<br>
> <br>
> I know there are "zero" values loc meaning some points where not binned, but that was always reflected in count too.<br>
> <br>
<br>
Let me return the question: is there any reason max(loc) should match size(count)? To me no, one is the size of the edges, another is the largest bin found. There is nothing to prevent user to pass the edges that cover the data wider than it really needs.<br>
<br>
Bruno