Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Index matrix to sparse matrix... saving memory

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 29 May, 2010 16:30:23

Message: 1 of 29

Latest problem....

I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.

S = [1 2 3 0 0 0 0 0 0
       4 5 6 0 0 0 0 0 0
       7 8 9 10 11 12 13 14 0
       15 16 17 18 19 20 21 22 23]

I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!

S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)

Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?

I am trying to figure out if I can use Bruno Luong's Sparse Sub Access, but I don't have su access to the server I am using to test it out...

Thanks in advance

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 29 May, 2010 16:43:05

Message: 2 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...
> Latest problem....
>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
>
> S = [1 2 3 0 0 0 0 0 0
> 4 5 6 0 0 0 0 0 0
> 7 8 9 10 11 12 13 14 0
> 15 16 17 18 19 20 21 22 23]
>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
>
> S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)
>
> Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?

The following command will return the non-zeros elements of matrix:

nonzeros(S_s(row,:))

I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.

AFAICS, you don not need my sparse access tool.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 29 May, 2010 16:58:05

Message: 3 of 29

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <htrg6p$2h3$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...
> > Latest problem....
> >
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
> >
> > S = [1 2 3 0 0 0 0 0 0
> > 4 5 6 0 0 0 0 0 0
> > 7 8 9 10 11 12 13 14 0
> > 15 16 17 18 19 20 21 22 23]
> >
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
> >
> > S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)
> >
> > Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?
>
> The following command will return the non-zeros elements of matrix:
>
> nonzeros(S_s(row,:))
>
> I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.
>
> AFAICS, you don not need my sparse access tool.
>
> Bruno

Bruno,

That is a solid!! And you are right, transpose and it screams.

Thank you again!!

Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Matt J

Date: 29 May, 2010 17:49:22

Message: 4 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...
> Latest problem....
>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
>
> S = [1 2 3 0 0 0 0 0 0
> 4 5 6 0 0 0 0 0 0
> 7 8 9 10 11 12 13 14 0
> 15 16 17 18 19 20 21 22 23]
>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
==================

Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?

That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 29 May, 2010 18:10:21

Message: 5 of 29

"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrk32$2vm$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrfev$fo7$1@fred.mathworks.com>...
> > Latest problem....
> >
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
> >
> > S = [1 2 3 0 0 0 0 0 0
> > 4 5 6 0 0 0 0 0 0
> > 7 8 9 10 11 12 13 14 0
> > 15 16 17 18 19 20 21 22 23]
> >
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
> ==================
>
> Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?
>
> That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.

Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).

I'll think about it, maybe I can apply this technique... Thanks again!

Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Matt J

Date: 29 May, 2010 18:28:05

Message: 6 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...
 
> Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).
>
> I'll think about it, maybe I can apply this technique... Thanks again!
======

Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.

 

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 29 May, 2010 21:56:05

Message: 7 of 29

"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrmbl$nn6$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...
>
> > Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).
> >
> > I'll think about it, maybe I can apply this technique... Thanks again!
> ======
>
> Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.
>
>

I have two questions concerning the above and something else that goes to the above

1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:

for v=1:length(r_bin)
index_temp = find( r_3d >= rL(v) & r_3d <rH(v));
index_range(v,1:length(index_temp)) = index_temp;
clear index_temp;
end

The reason I am asking is because I don't really know how many points there will be in each vector index_temp... so I end up not being able to allocate all the space I would like in index_range... can I do this with a sparse matrix, or should I just do what I am doing here and make index_range a sparse matrix afterwords? (rL and rH are the upper and lower bounds of the range bin)

2. Given what you see here in number 1 I don't understand how I could create a logical array to do the same thing since each vector indexes points to different parts within r_3d and other matrices I'm not showing... Matt could you give me a clue?

Thanks again

Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Matt J

Date: 29 May, 2010 23:48:08

Message: 8 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...
> "Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <htrmbl$nn6$1@fred.mathworks.com>...
> > "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htrlad$in3$1@fred.mathworks.com>...
> >
> > > Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).
> > >
> > > I'll think about it, maybe I can apply this technique... Thanks again!
> > ======
> >
> > Well, do think about it, because I don't see why any of the above matters. For any vector of linear indices idx, there is an equivalent vector of logical indices idx_logical, which can be used to do all the same indexing operations with the exact same syntax.
> >
> >
>
> I have two questions concerning the above and something else that goes to the above
>
> 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:
>
> for v=1:length(r_bin)
> index_temp = find( r_3d >= rL(v) & r_3d <rH(v));
> index_range(v,1:length(index_temp)) = index_temp;
> clear index_temp;
> end
>
> The reason I am asking is because I don't really know how many points there will be in each vector index_temp... so I end up not being able to allocate all the space I would like in index_range... can I do this with a sparse matrix, or should I just do what I am doing here and make index_range a sparse matrix afterwords? (rL and rH are the upper and lower bounds of the range bin)
>
> 2. Given what you see here in number 1 I don't understand how I could create a logical array to do the same thing since each vector indexes points to different parts within r_3d and other matrices I'm not showing... Matt could you give me a clue?
================

Combining all the recommendations in this thread (e.g., loading the indices column-wise instead of row-wise), I get to the following:


%preallocate
N=length(r_bin);
I=cell(1,N);
J=I;


%compute sparse-indices
for v=1:N

 I{v}=find( r_3d >= rL(v) & r_3d <rH(v) );

 J{v}=I{v};
   J{v}(:)=v;

end

index_range=sparse([I{:}], [J{:}],true); %final sparse logical matrix

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 30 May, 2010 09:07:06

Message: 9 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...
>
>
> 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:
>
> for v=1:length(r_bin)
> index_temp = find( r_3d >= rL(v) & r_3d <rH(v));
> index_range(v,1:length(index_temp)) = index_temp;
> clear index_temp;
> end
>

You should definitively learn how to use function HISTC and family.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 30 May, 2010 11:25:10

Message: 10 of 29

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <htt9rq$1s8$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hts2hl$3tk$1@fred.mathworks.com>...
> >
> >
> > 1. I am curious about the best way to build these sparse matrices. Right now I do an element by element comparison to find the values within a certain range:
> >
> > for v=1:length(r_bin)
> > index_temp = find( r_3d >= rL(v) & r_3d <rH(v));
> > index_range(v,1:length(index_temp)) = index_temp;
> > clear index_temp;
> > end
> >
>
> You should definitively learn how to use function HISTC and family.
>
> Bruno

Bruno,

I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?

In the code above I am doing the first step of the indexing, indexing in range. r_3d is the range of (r,theta,phi) to each point within a 3d cartesian space referenced to some point within the 3d space. Can you work with me on how I would go about utilizing these functions to solve such a problem?

Thank you for the continued help, You guys are truly a essential piece to the matlab community!

Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 30 May, 2010 12:02:04

Message: 11 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...

>
> Bruno,
>
> I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?

If problem address in this thread needs can be solved with the built-in function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.

The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple for-loops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 30 May, 2010 12:20:05

Message: 12 of 29

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <httk3s$hb0$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...
>
> >
> > Bruno,
> >
> > I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?
>
> If problem address in this thread needs can be solved with the built-in function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.
>
> The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple for-loops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.
>
> Bruno

Bruno,

I think you are right on with your statement "It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way." I will look back at the code you had posted before and think about this problem in a more abstract way.

Thanks again,

Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 8 Jun, 2010 19:47:04

Message: 13 of 29

Sorry to "double" post this (seems to fit here much better than the other thread)... but I think I am starting to see the light on this today... I haven't looked at it in awhile. for the case of binning in range, code shown above, I am using this:

r_bin = (bin_r/2:bin_r:bin_r_max*bin_r+bin_r/2);%[m]
[n,bin] = histc(r_3d(:),r_bin);

This gives me n and bin with all the information I need, I am playing with it to figure out how to extract the index values so I have a n x max(n) matrix. But can't I use your histcn function to do all the binning I need to do... r,theta, and phi... at once like this?

r_bin = (bin_r/2:bin_r:bin_r_max*bin_r+bin_r/2);%[m]
theta_bin = (bin_t/2:bin_t:bin_t_max*bin_size+bin_t/2);%[deg]
phi_bin = (bin_p/2:bin_p:bin_p_max*bin_p+bin_p/2);%[deg]

hugearray = [r_3d(:) theta_3d(:) phi_3d(:) ];
[count edges mid loc] = histcn(hugearray,r_bin,theta_bin,phi_bin);

where count and loc give me essentially the same thing as n and bin that I can then try and place into a sparse matrix to store for later use?

Thank you,

Anthony


"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <httl5l$ns3$1@fred.mathworks.com>...
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <httk3s$hb0$1@fred.mathworks.com>...
> > "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <htthum$sg$1@fred.mathworks.com>...
> >
> > >
> > > Bruno,
> > >
> > > I had downloaded your histcn function at one point to try and understand how it could be applied to my problem, but I must not fully understand it's abilities. To me, I would think you would need to work from the edge of a matrix to use the HISTC and HISTCN functions. Starting from an arbitrary point (r=0 in the above case could be in the center of the matrix), I don't understand how the function could be used to bin evenly spaced cartesian 3d space into bins defined by their spherical transforms?
> >
> > If problem address in this thread needs can be solved with the built-in function HISTC (one dimensional binning). You have to understand how it works. The way you are using FIND for the same purpose is very inefficient.
> >
> > The HISTCN I showed in other thread is an extension of HISTC. You don't need it here. And in the other thread I showed you the code that gives the rigorously identical result that with the double/triple for-loops in the first post. It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way. Think in a more abstract way of mapping and you'll be fine.
> >
> > Bruno
>
> Bruno,
>
> I think you are right on with your statement "It seems you are trying to visualize what HISTN does in term of geometry, and it gets you lost in the way." I will look back at the code you had posted before and think about this problem in a more abstract way.
>
> Thanks again,
>
> Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 8 Jun, 2010 20:25:20

Message: 14 of 29

First, this a an month-old thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the miss-use of sparse), the a lot time is waste.

Now it will be the last time to write something on this topic, then I'll move on. Here we go:

First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.

%%
function [ index_range ] = range_bin_sparse( r_4d, bin_size )
%RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size
% each radar is done individually and the ouput file is a sparse
% matrix
% [ range_index ] = range_bin_sparse( r_4d, bin_size )

%single range bin binning of a single radar
r_bin_max = ceil((((max(r_4d(:)))))/bin_size);
r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]
index_range = zeros(length(r_bin),250);

%filter variables
rH = r_bin + bin_size/2;
rL = r_bin - bin_size/2;
%find exceptable values for each range bin
for v=1:length(r_bin)
    index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here
    index_range(v,1:length(index_temp)) = index_temp;
end
index_range = sparse(index_range');

end

% Now here is the data I use to test
r_4d=peaks(50)+7; % + 7 to make data positive
bin_size = 0.1;
[ index_range ] = range_bin_sparse( r_4d, bin_size );

% Here is a way to use HISTC
nbins = ceil(max(r_4d(:))/bin_size);
edges = (0:nbins)*bin_size;
[~,bin] = histc(r_4d,edges);
bool = sparse(1:numel(bin), bin,true);

% Note the the logical sparse matrix BOOL is not exactly like range, but it contains the
% same information, if you want to find which index on the 50th bins, you can do this:

>> index_range(:,50)

ans =

   (1,1) 525
   (2,1) 526
   (3,1) 527
   (4,1) 528
   (5,1) 580
   (6,1) 826
   (7,1) 830
   (8,1) 1013
   (9,1) 1064
  (10,1) 1206
  (11,1) 1406
  (12,1) 1517

% Or this:

>> find(bool(:,50))

ans =

         525
         526
         527
         528
         580
         826
         830
        1013
        1064
        1206
        1406
        1517

% You see that they contain the same thing.

In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins

>> r_4d(bool(:,50))

ans =

    4.9948
    4.9059
    4.9003
    4.9737
    4.9775
    4.9397
    4.9658
    4.9965
    4.9447
    4.9068
    4.9602
    4.9365

% Check if they fall inside the bins

 >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)

ans =

     1
     1
     1
     1
     1
     1
     1
     1
     1
     1
     1
     1

>>

% Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 8 Jun, 2010 20:40:05

Message: 15 of 29

Thank you for your help, I apologize for my slow learning... you guys really are very good!

Anthony

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...
> First, this a an month-old thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the miss-use of sparse), the a lot time is waste.
>
> Now it will be the last time to write something on this topic, then I'll move on. Here we go:
>
> First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.
>
> %%
> function [ index_range ] = range_bin_sparse( r_4d, bin_size )
> %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size
> % each radar is done individually and the ouput file is a sparse
> % matrix
> % [ range_index ] = range_bin_sparse( r_4d, bin_size )
>
> %single range bin binning of a single radar
> r_bin_max = ceil((((max(r_4d(:)))))/bin_size);
> r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]
> index_range = zeros(length(r_bin),250);
>
> %filter variables
> rH = r_bin + bin_size/2;
> rL = r_bin - bin_size/2;
> %find exceptable values for each range bin
> for v=1:length(r_bin)
> index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here
> index_range(v,1:length(index_temp)) = index_temp;
> end
> index_range = sparse(index_range');
>
> end
>
> % Now here is the data I use to test
> r_4d=peaks(50)+7; % + 7 to make data positive
> bin_size = 0.1;
> [ index_range ] = range_bin_sparse( r_4d, bin_size );
>
> % Here is a way to use HISTC
> nbins = ceil(max(r_4d(:))/bin_size);
> edges = (0:nbins)*bin_size;
> [~,bin] = histc(r_4d,edges);
> bool = sparse(1:numel(bin), bin,true);
>
> % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the
> % same information, if you want to find which index on the 50th bins, you can do this:
>
> >> index_range(:,50)
>
> ans =
>
> (1,1) 525
> (2,1) 526
> (3,1) 527
> (4,1) 528
> (5,1) 580
> (6,1) 826
> (7,1) 830
> (8,1) 1013
> (9,1) 1064
> (10,1) 1206
> (11,1) 1406
> (12,1) 1517
>
> % Or this:
>
> >> find(bool(:,50))
>
> ans =
>
> 525
> 526
> 527
> 528
> 580
> 826
> 830
> 1013
> 1064
> 1206
> 1406
> 1517
>
> % You see that they contain the same thing.
>
> In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins
>
> >> r_4d(bool(:,50))
>
> ans =
>
> 4.9948
> 4.9059
> 4.9003
> 4.9737
> 4.9775
> 4.9397
> 4.9658
> 4.9965
> 4.9447
> 4.9068
> 4.9602
> 4.9365
>
> % Check if they fall inside the bins
>
> >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)
>
> ans =
>
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
>
> >>
>
> % Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 9 Jun, 2010 14:21:07

Message: 16 of 29

Thank you Matt and Bruno!! I apologize for not staying with this topic, but I have quite a bit on my plate... and I am taking your advice, and TRYING to learn. You have both been very helpful and I hope you won't give up on me yet!!

The reason I didn't stick with Matt's suggestion is I could never get values into the cells I and J, and then I got pulled from my work (circumstances that I do not wish to bring up).

I am working with your suggestions and trying to apply the code to a 3d matrix, but first I started playing with your code to see what happens if there are NaN values within r and I run into a problem that I thought I could explain, the error I receive is:

??? Error using ==> sparse
Index into matrix must be positive.

I realize the problem is that bin will have a 0 values due to the fact there is an out of bounds points... so I thought a remedy would be:

r_4d=peaks(50)+7; % + 7 to make data positive
r_4d(25:30,25:30) = NaN;
bin_size = 0.1;
%[ index_range ] = range_bin_sparse( r_4d, bin_size );

% Here is a way to use HISTC
nbins = 151;%ceil(max(r_4d(:))/bin_size);
r_bin = (0:nbins)*bin_size;
[null,bin] = histc(r_4d,r_bin);
bool = sparse(1:nnz(bin), nonzeros(bin),true);

As you notice I have modified r_4d(25,25) to be a NaN value and commented out index_range (this line also perplexes me). My final two changes are in the sparse command where I try and compensate for the fact that there are out of bounds points... this seems to mess things up, when I go to a much larger data set with a 3d matrix r_4d the code runs, the number of nnz points in bool matches the number of ~isnan(r_4d) so I know histc is still working... but when I index using bool the returned values are not correct.

for the above code:
without NaNs:
bool(:,50)

ans =

 (525,1) 1
 (526,1) 1
 (527,1) 1
 (528,1) 1
 (580,1) 1
 (826,1) 1
 (830,1) 1
(1013,1) 1
(1064,1) 1
(1206,1) 1
(1406,1) 1
(1517,1) 1

with NaNs:
bool(:,50)

ans =

 (525,1) 1
 (526,1) 1
 (527,1) 1
 (528,1) 1
 (580,1) 1
 (826,1) 1
 (830,1) 1
(1013,1) 1
(1064,1) 1
(1206,1) 1
(1382,1) 1
(1481,1) 1

Can you please give me a little more advice on this?

Thank you

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...
> First, this a an month-old thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the miss-use of sparse), the a lot time is waste.
>
> Now it will be the last time to write something on this topic, then I'll move on. Here we go:
>
> First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.
>
> %%
> function [ index_range ] = range_bin_sparse( r_4d, bin_size )
> %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size
> % each radar is done individually and the ouput file is a sparse
> % matrix
> % [ range_index ] = range_bin_sparse( r_4d, bin_size )
>
> %single range bin binning of a single radar
> r_bin_max = ceil((((max(r_4d(:)))))/bin_size);
> r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]
> index_range = zeros(length(r_bin),250);
>
> %filter variables
> rH = r_bin + bin_size/2;
> rL = r_bin - bin_size/2;
> %find exceptable values for each range bin
> for v=1:length(r_bin)
> index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here
> index_range(v,1:length(index_temp)) = index_temp;
> end
> index_range = sparse(index_range');
>
> end
>
> % Now here is the data I use to test
> r_4d=peaks(50)+7; % + 7 to make data positive
> bin_size = 0.1;
> [ index_range ] = range_bin_sparse( r_4d, bin_size );
>
> % Here is a way to use HISTC
> nbins = ceil(max(r_4d(:))/bin_size);
> edges = (0:nbins)*bin_size;
> [~,bin] = histc(r_4d,edges);
> bool = sparse(1:numel(bin), bin,true);
>
> % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the
> % same information, if you want to find which index on the 50th bins, you can do this:
>
> >> index_range(:,50)
>
> ans =
>
> (1,1) 525
> (2,1) 526
> (3,1) 527
> (4,1) 528
> (5,1) 580
> (6,1) 826
> (7,1) 830
> (8,1) 1013
> (9,1) 1064
> (10,1) 1206
> (11,1) 1406
> (12,1) 1517
>
> % Or this:
>
> >> find(bool(:,50))
>
> ans =
>
> 525
> 526
> 527
> 528
> 580
> 826
> 830
> 1013
> 1064
> 1206
> 1406
> 1517
>
> % You see that they contain the same thing.
>
> In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins
>
> >> r_4d(bool(:,50))
>
> ans =
>
> 4.9948
> 4.9059
> 4.9003
> 4.9737
> 4.9775
> 4.9397
> 4.9658
> 4.9965
> 4.9447
> 4.9068
> 4.9602
> 4.9365
>
> % Check if they fall inside the bins
>
> >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)
>
> ans =
>
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
> 1
>
> >>
>
> % Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 9 Jun, 2010 14:59:08

Message: 17 of 29

So I think a couple lines fixed the problem:

%%%%%%
bin_size = 100;
r_bin_max = ceil((((nanmax(double(r_4d(:))))))/bin_size);
r_bin = (0:bin_size:r_bin_max*bin_size);%[m]
[null,bin] = histc(r_4d(:),r_bin);
bin(bin==0)=max(bin(:))+1;
bool = sparse(1:numel(bin), (bin),true);
bool = bool(:,1:end-1);

I added the r_4d(:) so I can work with 3d matrices, and I took all the values of bin =0 and changed their value to something outside of the actual r_4d matrix index values and then lopped them off at the end.

Again, Thank you for all of your help. Now to apply this to r,theta, and phi... I think i am going to do this sequentially.

Anthony


"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <huo80j$c5n$1@fred.mathworks.com>...
> Thank you Matt and Bruno!! I apologize for not staying with this topic, but I have quite a bit on my plate... and I am taking your advice, and TRYING to learn. You have both been very helpful and I hope you won't give up on me yet!!
>
> The reason I didn't stick with Matt's suggestion is I could never get values into the cells I and J, and then I got pulled from my work (circumstances that I do not wish to bring up).
>
> I am working with your suggestions and trying to apply the code to a 3d matrix, but first I started playing with your code to see what happens if there are NaN values within r and I run into a problem that I thought I could explain, the error I receive is:
>
> ??? Error using ==> sparse
> Index into matrix must be positive.
>
> I realize the problem is that bin will have a 0 values due to the fact there is an out of bounds points... so I thought a remedy would be:
>
> r_4d=peaks(50)+7; % + 7 to make data positive
> r_4d(25:30,25:30) = NaN;
> bin_size = 0.1;
> %[ index_range ] = range_bin_sparse( r_4d, bin_size );
>
> % Here is a way to use HISTC
> nbins = 151;%ceil(max(r_4d(:))/bin_size);
> r_bin = (0:nbins)*bin_size;
> [null,bin] = histc(r_4d,r_bin);
> bool = sparse(1:nnz(bin), nonzeros(bin),true);
>
> As you notice I have modified r_4d(25,25) to be a NaN value and commented out index_range (this line also perplexes me). My final two changes are in the sparse command where I try and compensate for the fact that there are out of bounds points... this seems to mess things up, when I go to a much larger data set with a 3d matrix r_4d the code runs, the number of nnz points in bool matches the number of ~isnan(r_4d) so I know histc is still working... but when I index using bool the returned values are not correct.
>
> for the above code:
> without NaNs:
> bool(:,50)
>
> ans =
>
> (525,1) 1
> (526,1) 1
> (527,1) 1
> (528,1) 1
> (580,1) 1
> (826,1) 1
> (830,1) 1
> (1013,1) 1
> (1064,1) 1
> (1206,1) 1
> (1406,1) 1
> (1517,1) 1
>
> with NaNs:
> bool(:,50)
>
> ans =
>
> (525,1) 1
> (526,1) 1
> (527,1) 1
> (528,1) 1
> (580,1) 1
> (826,1) 1
> (830,1) 1
> (1013,1) 1
> (1064,1) 1
> (1206,1) 1
> (1382,1) 1
> (1481,1) 1
>
> Can you please give me a little more advice on this?
>
> Thank you
>
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hum8vg$65s$1@fred.mathworks.com>...
> > First, this a an month-old thread. I'm sorry but if you haven't not pay enough attention to what has been suggested for month (including Matt's remark about the miss-use of sparse), the a lot time is waste.
> >
> > Now it will be the last time to write something on this topic, then I'll move on. Here we go:
> >
> > First I change your original function by removing the garbage inside, and also change one comparison test (from > to >=) to match HISTC.
> >
> > %%
> > function [ index_range ] = range_bin_sparse( r_4d, bin_size )
> > %RANGE_BIN Identifies the index values in r_4d that are within a bin of a given size
> > % each radar is done individually and the ouput file is a sparse
> > % matrix
> > % [ range_index ] = range_bin_sparse( r_4d, bin_size )
> >
> > %single range bin binning of a single radar
> > r_bin_max = ceil((((max(r_4d(:)))))/bin_size);
> > r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]
> > index_range = zeros(length(r_bin),250);
> >
> > %filter variables
> > rH = r_bin + bin_size/2;
> > rL = r_bin - bin_size/2;
> > %find exceptable values for each range bin
> > for v=1:length(r_bin)
> > index_temp = find((r_4d) >= rL(v) & (r_4d) <rH(v)); % change here
> > index_range(v,1:length(index_temp)) = index_temp;
> > end
> > index_range = sparse(index_range');
> >
> > end
> >
> > % Now here is the data I use to test
> > r_4d=peaks(50)+7; % + 7 to make data positive
> > bin_size = 0.1;
> > [ index_range ] = range_bin_sparse( r_4d, bin_size );
> >
> > % Here is a way to use HISTC
> > nbins = ceil(max(r_4d(:))/bin_size);
> > edges = (0:nbins)*bin_size;
> > [~,bin] = histc(r_4d,edges);
> > bool = sparse(1:numel(bin), bin,true);
> >
> > % Note the the logical sparse matrix BOOL is not exactly like range, but it contains the
> > % same information, if you want to find which index on the 50th bins, you can do this:
> >
> > >> index_range(:,50)
> >
> > ans =
> >
> > (1,1) 525
> > (2,1) 526
> > (3,1) 527
> > (4,1) 528
> > (5,1) 580
> > (6,1) 826
> > (7,1) 830
> > (8,1) 1013
> > (9,1) 1064
> > (10,1) 1206
> > (11,1) 1406
> > (12,1) 1517
> >
> > % Or this:
> >
> > >> find(bool(:,50))
> >
> > ans =
> >
> > 525
> > 526
> > 527
> > 528
> > 580
> > 826
> > 830
> > 1013
> > 1064
> > 1206
> > 1406
> > 1517
> >
> > % You see that they contain the same thing.
> >
> > In practice, next calculation does not need the FIND command, because BOOL can be used as matrix for LOGICAL indexing, that's what Matt suggest you. For example figure out values of thet array r_4d() falling the 50th bins
> >
> > >> r_4d(bool(:,50))
> >
> > ans =
> >
> > 4.9948
> > 4.9059
> > 4.9003
> > 4.9737
> > 4.9775
> > 4.9397
> > 4.9658
> > 4.9965
> > 4.9447
> > 4.9068
> > 4.9602
> > 4.9365
> >
> > % Check if they fall inside the bins
> >
> > >> r_4d(bool(:,50))>=edges(50) & r_4d(bool(:,50))<edges(51)
> >
> > ans =
> >
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> > 1
> >
> > >>
> >
> > % Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 9 Jun, 2010 21:58:20

Message: 18 of 29

I feel like I am sooo close....

My issue is with finding the intersection between 3 sparse logical matrices

idxr, idxa, and idxe are all sparse logical matrices. The r, a and e correspond to range, azimuth and elevation. With the help of Bruno and Matt I find them like this:

r_bin_max = ceil((((nanmax(r_4d(:)))))/bin_size);
r_bin = (bin_size/2:bin_size:r_bin_max*bin_size+bin_size/2);%[m]
az_bin = 0:beamwidth:360;%[deg]
el_bin = 0:beamwidth:thetae_max;%[deg]
%
[null,bin] = histc(r_4d(:),r_bin);
clear null
bin(bin==0)=max(bin(:))+1;
idxr = sparse(1:numel(bin), (bin),true);
idxr = idxr(:,1:end-1);
[null,ebin] = histc(el(:),el_bin);
clear null
ebin(ebin==0)=max(ebin(:))+1;
idxe = sparse(1:numel(ebin), (ebin),true);
idxe = idxe(:,1:end-1);
[null,abin] = histc(az(:),az_bin);
clear null
abin(abin==0)=max(abin(:))+1;
idxa = sparse(1:numel(abin), (abin),true);
idxa = idxa(:,1:end-1);

%% I was going to use histcn but this takes less than 0.5sec to complete with histc, the histc calls take ~0.05sec, a similar histcn call would be ~0.15sec.

This is infinitely faster than the way I was doing it before, but I have the problem of finding the intersection between them to create the index representative of a 3d sampling in r,theta,phi... the way I do it now is:

idxbr =( zeros(size(idxr,2)*size(idxa,2)*size(idxe,2),100));

counter = 0;
for iir = 1:size(idxr,2)
    for iia = 1:size(idxa,2)
        for iie = 1:size(idxe,2)
            idxrae = intersect(find(idxr(:,iir)),intersect(find(idxa(:,iia)),find(idxe(:,iie))));
            counter = counter+1;
            idxbr(1:length(idxrae),counter) = idxrae;
            clear idxrae
        end%***end of beam position second loop
    end%***end of range first loop
end
idxbr = sparse(idxbr);
end%function end

This not only takes an extremely long time for large idx* arrays, but I loss the excellent logical sparse arrays Bruno and Matt taught me how to make.

if it is any consolation I know everything I need is in rbin, ebin, and abin... I can see it when I print the values out, the bins are right there to use.

Can I get rid of these for loops?

Thanks, Anthony

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 10 Jun, 2010 09:28:03

Message: 19 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hup2ps$rmb$1@fred.mathworks.com>...

>
> Can I get rid of these for loops?

For multi-dimensional histogram use Fex submission HISTCN.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 10 Jun, 2010 18:58:04

Message: 20 of 29

Bruno or anyone that has looked through the thread,

I'm beating myself up here... I just can't see through to the end. Histcn returns the same thing Histc does is return one output "loc" rather than multiple outputs of "bin." If I cat the bins

[bin1 bin2 bin3] == [ loc ]

Where I hit a wall is making use of loc. Loc has all of my bin locations, which I need and it does it very quickly, but I need to relate each column of "loc" to one another, compare every combination of bin value. This iterative comparison is what slows me down, and what I have been stuck on conceptually, which was what originally slowed down my understanding of your function histcn.

Does anyone know of a function that will compress these for loop after I have used histcn to find the bin values? The for loops step through the bin values of each column of "loc" returning the correct index values of "loc".

%I previously used histcn to create "loc"
idxbr =( zeros(length(azm2)*length(el2)*length(rb2),100));
counter = 0;

for iia = 1:max(loc(:,1)) %stepping through loc(:,1)
    for iie = 1:max(loc(:,2)) %stepping through loc(:,2)
        for iir = 1:max(loc(:,3)) %stepping through loc(:,3)
            counter = counter+1;
            tempfind = find(loc2(:,1)==iia & loc2(:,2)==iie & loc2(:,3) == iir); %find all values corresponding to loc = [1 1 1] to loc = [max(loc(:,1)) max(loc(:,2)) max(loc(:,3))] and everything in between.
            idxbr(1:length(tempfind),counter) = tempfind(:); %store the indexes
             clear tempfind
        end
    end
end

My buddy does this kind of lookup in SQL databases, but I don't know how to implement it in Matlab... I would assume there is a function, trick, or something I missed in histcn that someone can hint to? It I just had to find all points that sit in bin = 1, 2, 3, .... then this would be very simple, but because I have to look at each combination of bins it makes it tricky.

I realize this is a long thread but I hope someone reads this and can give me a hint!!

It is greatly appreciated.

Anthony

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <huqb73$q0d$1@fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hup2ps$rmb$1@fred.mathworks.com>...
>
> >
> > Can I get rid of these for loops?
>
> For multi-dimensional histogram use Fex submission HISTCN.
>
> Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 10 Jun, 2010 19:18:04

Message: 21 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurcjs$h31$1@fred.mathworks.com>...
> Bruno or anyone that has looked through the thread,
>
> I'm beating myself up here... I just can't see through to the end. Histcn returns the same thing Histc does is return one output "loc" rather than multiple outputs of "bin." If I cat the bins
>
> [bin1 bin2 bin3] == [ loc ]
>
> Where I hit a wall is making use of loc. Loc has all of my bin locations, which I need and it does it very quickly, but I need to relate each column of "loc" to one another, compare every combination of bin value. This iterative comparison is what slows me down, and what I have been stuck on conceptually, which was what originally slowed down my understanding of your function histcn.
>
> Does anyone know of a function that will compress these for loop after I have used histcn to find the bin values? The for loops step through the bin values of each column of "loc" returning the correct index values of "loc".

I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTC. In this thread, the ACCUMARRAY does carry out the sum (because that was originally asked), but you can adapt it to your specific need, including find all the indexes that belong to the n-dimensional bins.

Again, that was answered in Apr 27th, 2010.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 10 Jun, 2010 19:38:05

Message: 22 of 29

I wrote:
>
> I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...

Sorry, should read "HISTCN"

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 10 Jun, 2010 19:57:04

Message: 23 of 29

I understood, and appreciate your help. accumarray it is!!

I'll post my solution when I figure this function out.

Anthony

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...
> I wrote:
> >
> > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...
>
> Sorry, should read "HISTCN"
>
> Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 13 Jun, 2010 14:45:06

Message: 24 of 29

Bruno... I can't believe I missed the funcitonality of histcn and accumarray!! This definitely gets me closer to my ultimate result.

The below countermatrix returns the number of points within each bin given by histcn... This works extremely fast even for very large 3d matrices. I ran into a bit of a problem with "out of range" points, but dealt with these by tossing them in outer bins and taking them off of the countermatrix in the final steps. (code is at the bottom of this post).

This matrix isn't exactly what I was looking for though...

1. I'm curious if there is a way to collect the index values, corresponding to the original val3d matrix, into bin vectors. What I would like is a vector of index values for each point within countermatrix, at this point the structure could be arbitrary but optimally it could be indexed by each bin value... cell arrays of cell array, rbincell{tbincell{pbincell{[idx_vals for countermatrix(r,t,p)]}}} (This is my first time working with the ideas of cells). Keeping the index values accumulated seems doable with accumarray because it "touches" each one of these values... the array being operated on by accumarray would just be the index values corresponding to the points within val3d, rather than val3d itself

2. Given the above I would also like to fill the original matrix val3d with the accumulated results, for example if val3d(i,j,k) fell into the sum corresponding to countermatrix(r,t,p), then we would put that sum in countermatrix(r,t,p) back into val3d(i,j,k), and all other points that where used for that particular sum. I do this iteratively now after finding the matrix mentioned in 1 above... but again this is a for loop that I want to get rid of.

here is the code that creates countermatrix, in this case val3d is an array of ones and we should find that count and countermatrix are equal:

%variables
x = 1e3.*(1:.1:2);
y = 1e3.*(0:.1:1);
z = 1e3.*(0:.1:1);

[x3d,y3d,z3d] = meshgrid(x,y,z);
[t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);
%vector to accumulate from
val3d = ones(size(x3d));

%maximum values of the bins
maxt = max(t3d(:));
maxp = max(p3d(:))-(pi/180);
maxr = max(r3d(:));
%bins
tbin = linspace(0,maxt,6);
pbin = linspace(0,maxp,6);
rbin = linspace(1e3,maxr,6);
%histogram
data = [r3d(:) t3d(:) p3d(:)];

[count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);

% temp = loc(:,1);temp(temp ==0)=max(temp)+1;loc(:,1) = temp;

%place "out of range" values at the max bin for each rbin,tbin,pbin
nir = zeros(1,3);
for oor = 1:3
 temp = loc(:,oor);
 if any(temp==0) %if there are out of range values
 temp(temp ==0)=max(temp)+1;
 loc(:,oor) = temp;
 clear temp
 nir(oor)=1;
 end
end

countermatrix_accum = accumarray(loc,val3d(:));

%remove "out of range" accumulated bins
if nir(1)
    countermatrix_accum = countermatrix_accum(1:end-1,:,:);
end
if nir(2)
    countermatrix_accum = countermatrix_accum(:,1:end-1,:);
end
if nir(3)
    countermatrix_accum = countermatrix_accum(:,:,1:end-1);
end

isequal(count,countermatrix_accum)


"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurg2g$5kt$1@fred.mathworks.com>...
> I understood, and appreciate your help. accumarray it is!!
>
> I'll post my solution when I figure this function out.
>
> Anthony
>
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...
> > I wrote:
> > >
> > > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...
> >
> > Sorry, should read "HISTCN"
> >
> > Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 14 Jun, 2010 01:17:04

Message: 25 of 29

Thank you very much Bruno, and others, for giving your advise...

I had all of the resources to accomplish number 2... haven't figured out number 1 yet. Here is the code I am working with now, in case you are curious, i'm sure it isn't optimized:

%variables
x = 1e3.*(1:.1:2);
y = 1e3.*(0:.1:1);
z = 1e3.*(0:.1:1);

[x3d,y3d,z3d] = meshgrid(x,y,z);
[t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);
%vector to accumulate from
val3d = ones(size(x3d));

%maximum values of the bins
maxt = max(t3d(:));
maxp = max(p3d(:))-(pi/180);
maxr = max(r3d(:));
%bins
tbin = linspace(0,maxt,6);
pbin = linspace(0,maxp,6);
rbin = linspace(1e3,maxr,6);
%histogram
data = [r3d(:) t3d(:) p3d(:)];

[count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);

%place "out of range" values at the max bin for each rbin,tbin,pbin
nir = zeros(1,3);
for oor = 1:3
 temp = loc(:,oor);
 if any(temp==0) %if there are out of range values
 temp(temp ==0)=max(temp)+1;
 loc(:,oor) = temp;
 clear temp
 nir(oor)=1;
 end
end

countermatrix_accum = accumarray(loc,val3d(:));

%set countermatrix "out of range" values to zero
if nir(1)
    countermatrix_accum(end,:,:) = 0;
end
if nir(2)
    countermatrix_accum(:,end,:) = 0;
end
if nir(3)
    countermatrix_accum(:,:,end) = 0;
end

%g will be the number of points at each common grid point
gidx = sub2ind(size(countermatrix_accum),loc(:,1),loc(:,2),loc(:,3));

g = countermatrix_accum(gidx);

%reshape g to get our matrix
newval3d = reshape(g,size(val3d));

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hv2qti$j1l$1@fred.mathworks.com>...
> Bruno... I can't believe I missed the funcitonality of histcn and accumarray!! This definitely gets me closer to my ultimate result.
>
> The below countermatrix returns the number of points within each bin given by histcn... This works extremely fast even for very large 3d matrices. I ran into a bit of a problem with "out of range" points, but dealt with these by tossing them in outer bins and taking them off of the countermatrix in the final steps. (code is at the bottom of this post).
>
> This matrix isn't exactly what I was looking for though...
>
> 1. I'm curious if there is a way to collect the index values, corresponding to the original val3d matrix, into bin vectors. What I would like is a vector of index values for each point within countermatrix, at this point the structure could be arbitrary but optimally it could be indexed by each bin value... cell arrays of cell array, rbincell{tbincell{pbincell{[idx_vals for countermatrix(r,t,p)]}}} (This is my first time working with the ideas of cells). Keeping the index values accumulated seems doable with accumarray because it "touches" each one of these values... the array being operated on by accumarray would just be the index values corresponding to the points within val3d, rather than val3d itself
>
> 2. Given the above I would also like to fill the original matrix val3d with the accumulated results, for example if val3d(i,j,k) fell into the sum corresponding to countermatrix(r,t,p), then we would put that sum in countermatrix(r,t,p) back into val3d(i,j,k), and all other points that where used for that particular sum. I do this iteratively now after finding the matrix mentioned in 1 above... but again this is a for loop that I want to get rid of.
>
> here is the code that creates countermatrix, in this case val3d is an array of ones and we should find that count and countermatrix are equal:
>
> %variables
> x = 1e3.*(1:.1:2);
> y = 1e3.*(0:.1:1);
> z = 1e3.*(0:.1:1);
>
> [x3d,y3d,z3d] = meshgrid(x,y,z);
> [t3d,p3d,r3d] = cart2sph(x3d,y3d,z3d);
> %vector to accumulate from
> val3d = ones(size(x3d));
>
> %maximum values of the bins
> maxt = max(t3d(:));
> maxp = max(p3d(:))-(pi/180);
> maxr = max(r3d(:));
> %bins
> tbin = linspace(0,maxt,6);
> pbin = linspace(0,maxp,6);
> rbin = linspace(1e3,maxr,6);
> %histogram
> data = [r3d(:) t3d(:) p3d(:)];
>
> [count,edges,mid,loc] = histcn(data,rbin,tbin,pbin);
>
> % temp = loc(:,1);temp(temp ==0)=max(temp)+1;loc(:,1) = temp;
>
> %place "out of range" values at the max bin for each rbin,tbin,pbin
> nir = zeros(1,3);
> for oor = 1:3
> temp = loc(:,oor);
> if any(temp==0) %if there are out of range values
> temp(temp ==0)=max(temp)+1;
> loc(:,oor) = temp;
> clear temp
> nir(oor)=1;
> end
> end
>
> countermatrix_accum = accumarray(loc,val3d(:));
>
> %remove "out of range" accumulated bins
> if nir(1)
> countermatrix_accum = countermatrix_accum(1:end-1,:,:);
> end
> if nir(2)
> countermatrix_accum = countermatrix_accum(:,1:end-1,:);
> end
> if nir(3)
> countermatrix_accum = countermatrix_accum(:,:,1:end-1);
> end
>
> isequal(count,countermatrix_accum)
>
>
> "Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <hurg2g$5kt$1@fred.mathworks.com>...
> > I understood, and appreciate your help. accumarray it is!!
> >
> > I'll post my solution when I figure this function out.
> >
> > Anthony
> >
> > "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hureus$mfm$1@fred.mathworks.com>...
> > > I wrote:
> > > >
> > > > I have show you on other thread (where it should belong the few last posts) the use of ACCUMARRAY after HISTCN ...
> > >
> > > Sorry, should read "HISTCN"
> > >
> > > Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 14 Jun, 2010 06:07:03

Message: 26 of 29

Take a look at example 10 of the doc of accumarray:
http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html

In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.

With something similar you can accomplish the task you want.

Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 14 Jun, 2010 10:20:04

Message: 27 of 29

Thank you Bruno... The extreme speed up I see using accumarray may negate the need to bin all of the index values for storage. But I will play with this more.

Anthony

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hv4gu7$ark$1@fred.mathworks.com>...
> Take a look at example 10 of the doc of accumarray:
> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html
>
> In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.
>
> With something similar you can accomplish the task you want.
>
> Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Anthony Hopf

Date: 15 Jul, 2010 13:20:21

Message: 28 of 29

Bruno,

Is there any reason that I would get different values for the output of histcn max(loc) and size(count)?

I have not had to look back at this part of my code until I started messing with using the count output instead of doing my accumarray on a array of ones to calculate the number of points in each bin.

for example
I am getting max(loc) = [6 4 9] and size(count) = [7 4 9]

I know there are "zero" values loc meaning some points where not binned, but that was always reflected in count too.

Hope all is well,

Anthony

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <hv4gu7$ark$1@fred.mathworks.com>...
> Take a look at example 10 of the doc of accumarray:
> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html
>
> In this example, the function handle "@(x) {x}" is used to group the "values" as cell per bin.
>
> With something similar you can accomplish the task you want.
>
> Bruno

Subject: Index matrix to sparse matrix... saving memory

From: Bruno Luong

Date: 16 Jul, 2010 06:54:05

Message: 29 of 29

"Anthony Hopf" <anthony.hopf@gmail.com> wrote in message <i1n1ul$mf0$1@fred.mathworks.com>...
> Bruno,
>
> Is there any reason that I would get different values for the output of histcn max(loc) and size(count)?
>
> I have not had to look back at this part of my code until I started messing with using the count output instead of doing my accumarray on a array of ones to calculate the number of points in each bin.
>
> for example
> I am getting max(loc) = [6 4 9] and size(count) = [7 4 9]
>
> I know there are "zero" values loc meaning some points where not binned, but that was always reflected in count too.
>

Let me return the question: is there any reason max(loc) should match size(count)? To me no, one is the size of the edges, another is the largest bin found. There is nothing to prevent user to pass the edges that cover the data wider than it really needs.

Bruno

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us