Thread Subject: Sparse function

Hi everyone!
I am processing data which contains millions n millions of Zeros. It's taking a lot of time to process. I was told that I can save myself time and memory space if I use "sparse", the Matlab help file didn't help much! Can anyone help please? any tips?
Thanks
Mario

"Mario Liverpool" <m.ragaa@liv.ac.uk> wrote in message <hdmi2h$i45$1@fred.mathworks.com>...
> Hi everyone!
> I am processing data which contains millions n millions of Zeros. It's taking a lot of time to process. I was told that I can save myself time and memory space if I use "sparse", the Matlab help file didn't help much! Can anyone help please? any tips?
> Thanks
> Mario

Well, for example, in the following, both A and A sparse represent the same matrix of zeros (3000x3000), but you can see that Asparse consumes 6000 times less memory:

>> A=zeros(3000);
>> Asparse=sparse(3000,3000);

>> whos A Asparse
  Name Size Bytes Class Attributes

  A 3000x3000 72000000 double
  Asparse 3000x3000 12016 double sparse

"Mario Liverpool" <m.ragaa@liv.ac.uk> wrote in message <hdmi2h$i45$1@fred.mathworks.com>...
> Hi everyone!
> I am processing data which contains millions n millions of Zeros. It's taking a lot of time to process. I was told that I can save myself time and memory space if I use "sparse", the Matlab help file didn't help much! Can anyone help please? any tips?
> Thanks
> Mario

Any tips would depend on what you want to do with the data. For some examples, see Loren's blog on creating sparse matrices, on march 1, 2008 if I recall

Thanks Tim for the help.
I am processing data as follow:
-First I read data (a vector of 500.000 row and 1 colomn)
-apply a threshold, which will leave me with the same size vector but with a lot of zeros.
-apply a grouping function, which will add up values of specefic rows and I end up with a vector of 360 rows and 1 colomn.
-I read sometimes more than 40 vectors for process!
How can the "sparce" option help me here?
thank u.
mario

> Any tips would depend on what you want to do with the data. For some examples, see Loren's blog on creating sparse matrices, on march 1, 2008 if I recall

"Mario Liverpool" <m.ragaa@liv.ac.uk> wrote in message <he5scl$5qs$1@fred.mathworks.com>...
> Thanks Tim for the help.
> I am processing data as follow:
> -First I read data (a vector of 500.000 row and 1 colomn)
> -apply a threshold, which will leave me with the same size vector but with a lot of zeros.
> -apply a grouping function, which will add up values of specefic rows and I end up with a vector of 360 rows and 1 colomn.
> -I read sometimes more than 40 vectors for process!
> How can the "sparce" option help me here?

It probably won't. And this is not even that terribly large
an amount of data.

Efficient coding will do far more for you than any usage
of sparse here. In fact, I'd wonder if a sparse vector
actually took more time (overall) to process for what
you are doing than a full vector. Don't forget the
amount of time required to convert it to sparse.

John

That's true John, when I tried to use sparse function, it took the program much longer to run. I am not using it anymore.
What do u mean by "efficient coding", I think I optimised my program to maximum, unless u have other tips? any general ideas?
Thanks
Mario

"John D'Errico" <woodchips@rochester.rr.com> wrote in message >
> It probably won't. And this is not even that terribly large
> an amount of data.
>
> Efficient coding will do far more for you than any usage
> of sparse here. In fact, I'd wonder if a sparse vector
> actually took more time (overall) to process for what
> you are doing than a full vector. Don't forget the
> amount of time required to convert it to sparse.
>
> John

"Mario Liverpool" <m.ragaa@liv.ac.uk> wrote in message <he6nmf$nbd$1@fred.mathworks.com>...
> That's true John, when I tried to use sparse function, it took the program much longer to run. I am not using it anymore.
> What do u mean by "efficient coding", I think I optimised my program to maximum, unless u have other tips? any general ideas?
========

Maybe read in all 40 data sets at once, rather than in 40 consecutive disk accesses/memory allocations.
Also, you might try reading them in and manipulating them as type single rather than double, if you can tolerate the loss of precision.

"Mario Liverpool" <m.ragaa@liv.ac.uk> wrote in message <he6nmf$nbd$1@fred.mathworks.com>...
> That's true John, when I tried to use sparse function, it took the program much longer to run. I am not using it anymore.

Sparse has advantages when you are solving systems
of equations. It has advantages when you have sparse
MATRICES. But to make a vector sparse, just because
it has a few zeros in it, is probably a waste of time.

For a sparse matrix, there are many problems that
simply could never be solved in a reasonable amount
of time (On less than a super computer) without
benefit of sparse matrices. But making your vector
sparse to save a few adds? Skip it.


> What do u mean by "efficient coding", I think I optimised my program to maximum, unless u have other tips? any general ideas?
> Thanks

You may THINK you have optimized your code, but
I'll bet it is not. I've returned multiple times to some
codes I've written, improving it significantly each
time. And I know matlab VERY well.

General ideas? Sorry. There are zillions of things
one can try, and I have no idea how well your code
is written, or how poorly.

An point to note are the coding contests run by TMW
on a periodic basis. See that the scores for these
contests often drops significantly over a period of
the few days of the contest. As dozens of people
tweak their codes for optimal performance, they
often find nifty tricks to get speed enhancements.

John

Thanks John for the advice, I tried few things on the code, and you are right again, codes can always re-written many times and improved better.
Mario



> You may THINK you have optimized your code, but
> I'll bet it is not. I've returned multiple times to some
> codes I've written, improving it significantly each
> time. And I know matlab VERY well.
>
> General ideas? Sorry. There are zillions of things
> one can try, and I have no idea how well your code
> is written, or how poorly.
>
> An point to note are the coding contests run by TMW
> on a periodic basis. See that the scores for these
> contests often drops significantly over a period of
> the few days of the contest. As dozens of people
> tweak their codes for optimal performance, they
> often find nifty tricks to get speed enhancements.
>
> John