Asked by Yuzixuan Zhu
on 13 Sep 2018

I wish to create m = 10^5 sparse matrices of size n by n, say n = 10^4. I have been using

A = cell(m, 1);

for i = 1:m

row = ...; col = ...; val = ...; % here ... means some certain assignment in column vectors

A{i} = sparse(row, col, val, n, n);

end

But it is too slow. So I tried to use the types ndSparse (https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays) and sptensor (https://www.sandia.gov/~tgkolda/TensorToolbox/index-2.6.html). They do the job fast by creating m matrices all at once in 3d (n*n*m). It requires concatenating index and value vectors, where the speed is acceptable. However, I then need individual matrices for some operations that do NOT work on types ndSparse and sptensor. For example,

[R, p] = chol(A(:, :, i));

does not work. If I convert the object to Matlab sparse type as

[R, p] = chol(sparse(A(:, :, i)));

then it is even slower than creating A one by one in the for loop. Considering that Matlab does not support multidimensional sparse arrays (so I cannot reshape the abovementioned types into Matlab sparse tensor), how can I speed up creating m sparse matrices? Thank you!

Answer by Matt J
on 13 Sep 2018

Edited by Matt J
on 13 Sep 2018

Accepted Answer

Once you have A in ndSparse form, you can then split it into the cell array form you were originally trying to get using mat2cell:

Ar=sparse( reshape(A,n,m*n) );

Acell=mat2cell(Ar,n,ones(1,m)*n);

and then

[R, p] = chol(Acell{i});

Matt J
on 14 Sep 2018

Yuzixuan Zhu
on 15 Sep 2018

I've been trying to think of a way to not use them simultaneously but haven't found a solution. However I found a way to work around building big sparse matrix (thanks for reminding me that even empty big sparse matrix can take up a lot of memory):

[C, ~, ic] = unique([row{i}; col{i}]);

len_C = length(C);

len_ic2 = length(ic)/2;

Ai = sparse(ic, [ic((len_ic2 + 1):end); ic(1:len_ic2)], val{i}, len_C, len_C);

This works for all my operations. However, the "unique" function in the first line above takes even more time than before... Is there any way to make it faster?

Yuzixuan Zhu
on 16 Sep 2018

I came up with a way to not keep all matrices at the same time, and codes as in the 3rd comment in this answer seem to save 50% time for the large data. Thank you!

May I know what the proper way to include "ndSparse.m" in my software is? Is it enough to include "license.txt" and cite the webpage "https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays" in my document?

Sign in to comment.

Answer by Matt J
on 13 Sep 2018

Edited by Matt J
on 13 Sep 2018

Matt J
on 13 Sep 2018

How does it "not apply" to your application?

Making CHOL a method of ndSparse will not address the problem because the bottleneck would still be in accessing the sub-matrices A(:,:,i). Even with normal 2D sparse matrices, you are seeing that that is problem.

Matt J
on 13 Sep 2018

What are the densities of these matrices nnz(A)/numel(A) ?

Yuzixuan Zhu
on 13 Sep 2018

Very sparse -- less then 10 nonzero elements for each matrix.

Sign in to comment.

Answer by Christine Tobler
on 13 Sep 2018

Yuzixuan Zhu
on 14 Sep 2018

Sign in to comment.

Opportunities for recent engineering grads.

Apply Today
## 1 Comment

## Steven Lord (view profile)

Direct link to this comment:https://www.mathworks.com/matlabcentral/answers/418890-how-to-build-many-sparse-matrices#comment_610077

Sign in to comment.