Code covered by the BSD License  

Highlights from
DM Utils (data mining utils)

Be the first to rate this file! 18 Downloads (last 30 days) File Size: 8.7 KB File ID: #34598
image thumbnail

DM Utils (data mining utils)

by

 

15 Jan 2012 (Updated )

The tools for dealing with distance matrix, improving data mining capabilities

| Watch this File

File Information
Description

Utility m-files to improve distance matrix usage and computation:
- parallel distance matrix computation (pair_dist_par),
- parallel distance computation using shared memory model capable of maintaining really large matrices (pair_dist_spmd) - requires sharedmatrix to be compiled and in path,
- function for a comfortable referring to a vector form of a distance matrix (pseudo_squareform).
The out=pair_dist_par(X,fun,parameters) works somewhat similar to standard pdist. It requires Parallel Computing Toolbox (PCT) but the difference is that distance function (fun) works in row-vs-row manner (instead row-vs-submatrix in pdist) and it is a handler to a distance function of a form d=fun(x,y,params), where: params is a variable list of parameters, x and y are single row vectors it is DIFFERENT to the pdist.
One can provide or a handler to own distance function or a name of a built-in distance function, or (which is better due to performance) to use additional external function name2fun to get an anonymous function handler.
If there is no PCT or trouble with getting a workers pool then pair_dist_seq is used.

The function pair_dist_spmd it is intended for rather large distance matrices.
The calculation is done using SPMD model with shared memory (SM)
as IPC, therefore, it requires to have in path J. Dillon's sharedmatrix (or win SharedMemory). The function returns a handler to sharedmemory segment containing
results. The cache hit-ratio is improved by the interleaving.
To use the results You need to attach to the shared variableby the returned handler.

out = sharedmatrix('attach', out_hdl);
or
out = SharedMemory('attach', out_hdl);

remember also to remove unused variable (there is no grabage collector)
sharedmatrix('detach', out_hdl, out);
sharedmatrix('free', out_hdl);

The pseudo_squareform function allows you to get an access to a distance matrix as if it were in a square form without a need to convert it. Classic index operators allowed (coordinates, whole columns/rows or ranges with minor discrepancy for a colon operator).

Further improvements will come...

P.S.
Due to cumbersome compilation of sharedmatrix win32 and lin64 mex files are also included.
P.P.S.
I have been ordered by Mathworks staff to exclude mexfiles from the contribution. If anyone is interested please contact me i can provide win32 and lin64 versions of shared matrix.

Acknowledgements

Sharedmatrix inspired this file.

Required Products Parallel Computing Toolbox
MATLAB release MATLAB 7.8 (R2009a)
Other requirements Parallel Computing Toolbox (suggested)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Updates
17 May 2013

Shared memory function included

20 May 2013

a file correction

06 Jun 2013

updated with spmd (and excluded mex files from submission as required by mtalabcentral staff)

29 Aug 2014

some typos corrected

Contact us