Mutual Information computation

A self-contained package for computing mutual information, joint/conditional probability, entropy

Description

A self-contained, cross-platform, package for computing mutual information, joint/conditional probability, entropy, and more. This package has also been used for general machine learning and data mining purposes such as feature selection, Bayesian network construction, signal processing, etc.

Another related package for minimal redundancy feature selection is also available at the Matlab Central exchange site, under the category of "Biotech and Pharmaceutical".

A simple demo is called demo_mi.m.

*** Please note that downloading or use of this package means acceptance of the license of this package. In short, this package is free to non-profit use but cannot be re-distributed in any form, including revised forms, without the explicit permission for the author, Hanchuan Peng. See readme file for further information. ***

• MATLAB release: MATLAB 7 (R14)

[Xiang Y]

great! easy to use.

[ali shrof]

 

[Peter Chow]

a nice piece of work, make my life much easier!

[Rui Liu]

very good!!

[Mike B]

 

[Fernando Diaz]

I need un function that is needed in mutualinfo (estpab)

[sumit maheshwari]

i want to write joint probabilty as a function or want to express as an equation....
please help me.......

[Dedy H]

Can you help me to find joint probability for multivariate data....???how can I contact with you???

[Usha Nair]

i want to normalise the mutualinfo to the range (0-1) for comparison of different time series. Could you please help me by proper suggestion for this. Is it possible to change the number of bins used for the histogram so that all the data sets can be analyzed using the same scale.

[Guenther Eibl]

I just needed the normalized MI which was easy to find and use

[ming ma]

very good

[Yan Jin]

Thank you for your source code! It's very good^^

[Bahaa Khalil]

Many thanks for your contribution, is it possible to adjust estpob file to be able to calculate mutual information for continuous data

[Hanchuan]

Quite some people ask for mex files for different operating system versions, such as the latest Windows, or Mac OS X 10.6 Snow Leopard, etc. Please note that the mex functions can simply be re-generated by running the "makeosmex" command when you enter the folder of this toolbox (I assume you know how to set up mex compiling environment). For Windows, I use mingw to compile.

[Carlos Guerrero Mosquera]

I can't compiler the file estmutualinfo.cpp.
estmutualinfo.cpp
estmutualinfo.cpp(42) : warning C4267: 'initializing' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(43) : warning C4267: 'initializing' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(55) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(58) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(65) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(67) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(120) : error C2668: 'log' : ambiguous call to overloaded function
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(567): could be 'long double log(long double)'
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(519): or 'float log(float)'
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(121): or 'double log(double)'
        while trying to match the argument list '(int)'

[Raphael]

I am trying to compile mex files on Unix with Matlab 7.7.0(2008b) using makeosmex.m. I get the following error messages:
building mex(dll) of estcondentropy.cpp
estcondentropy.cpp:19:23: error: miinclude.h: No such file or directory
estcondentropy.cpp:21: error: ?mxArray? has not been declared
estcondentropy.cpp:21: error: expected ?,? or ?...? before ?*? token
estcondentropy.cpp:21: error: ISO C++ forbids declaration of ?mxArray? with no type
estcondentropy.cpp: In function ?void mexFunction(int, int**, int, int)?:
estcondentropy.cpp:24: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:26: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:32: error: ?prhs? was not declared in this scope
estcondentropy.cpp:32: error: ?mxGetPr? was not declared in this scope
estcondentropy.cpp:33: error: ?mxGetM? was not declared in this scope
estcondentropy.cpp:34: error: ?mxGetN? was not declared in this scope
estcondentropy.cpp:41: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:59: error: ?log? was not declared in this scope
estcondentropy.cpp:65: error: ?log? was not declared in this scope
estcondentropy.cpp:67: error: ?mxREAL? was not declared in this scope
estcondentropy.cpp:67: error: ?mxCreateDoubleMatrix? was not declared in this scope

Suggestions?!

[Benjamin Mack]

Hello,
I try to use this programm but I am totally new to matlab (but have experience in R). When I call the function
mrmr_mid_d (variables, response, 25)
the following error occurs:
??? Undefined function or method 'mrmr_mid_d' for input arguments of type 'double'.
The same occures when I convert my data in int8 or single.

something similar happens when I try to use the demo_mi.m :

mutualinfo(a,b)
??? Undefined function or method 'estpab' for input arguments of type 'double'.
Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

I think my problem is very a very basic one because I do not know anything about Matlab.
Could anybody please help me. It is very important for me to use this programm and the online version does not work with my data set because it is too large.

Thank you very much in advance,
Ben
 

[Omid Aghazadeh]

I'm sure the log errors are solved by now. It is obvious that you need to add a .0 to the log(2) to solve the problem if you get that error. (change log(2) to log(2.0)).

[chetana nayaka]

thank u for the toolbox...

[wang]

Hi, Prof Peng,

I still met one problem when I compile "estcondentropy.cpp" by run "mex estcondentropy.cpp" after I changed log(2) to log(2.0).

The other files can be compiled successfully. Can you help me for compiling "estcondentropy.cpp"?

Thank you!
Simbest Wang

[wang]

I find the problem.
define "pabhei" "pabwid" with "long" is ok.
about line 32-34 in "estcondentropy.cpp" .

Is it right?

[Marie]

When I run the program, I have the error "Invalid mex-file: the ..........estentropy.dll file isn't a valid win32 application".
Could somebody explain to me what is the problem?
I am using the matlab version (R2009) on a windows machine.
Thank you very much!

[wang]

Hi, Marie, have you compiled the estentropy.cpp on your computer?

[Mihály]

Hi,
I re-compiled the cpp files with mex to match te current lib versions, but when I compute a mutual information, I always get 0. Did anybody encounter this problem? (I use Linux)

Thanks.

[Nanye]

I was using mutualinfo() and found the following errors

pure virtual method called
terminate called without an active exception
mutualinfo:778(<D,M> <D,C>):(46912990672832, 2433) -> (<T,T>)

 Any hint? Thanks!

[Jong Rah]

 

[Vishvas Vasuki]

Tried mutual information computation code. Works well!

[Ali Farzan]

I have the same problem as Benjamin Mack.
How can I fix it?

[Md. Ali Hossain]

Hello, I o not understand the parameter d and f clearly. What will be the value of d? is it individual band for a hyperspectral image, and what will f then? Please inform me.

Thanks
Ali

[Md. Ali Hossain]

Hello, I do not understand the parameter d and f clearly. What will be the value of d? is it individual band for a hyperspectral image, and what will f then? Please inform me.Its better if one can send me with some example data set.

Thanks
Ali

[Jochen Kumm]

I am running matlab on mac osx 10.4 (matlab 7.7.0 R2008b) and also on the unix server (matlab 7.11.0 R2010b), would you please tell me how I can compile the files on both of these machines.

I could run makeosmex.m on the unix machine without any error but when I tried to run demo_mi.m, it gave me the following error (I actually replaced log(2) with log(2.0) in estmutualinfo.cpp as well):

??? Undefined function or method 'estpab' for input arguments of
type 'double'.

Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> demo_mi at 25
mutualinfo(a,b)
 

On mac osx I even could not run makeosmex.m, it gave me the following error:

building mex(dll) of estcondentropy.cpp
  mex(list(i).name);
/Applications/MATLAB_R2008b.app/bin/mex: line 1026: gcc-4.0: command not found
/Applications/MATLAB_R2008b.app/bin/mex: line 1005: gcc-4.0: command not found
/Applications/MATLAB_R2008b.app/bin/mex: line 1: g++-4.0: command not found

    mex: compile of ' "estcondentropy.cpp"' failed.

??? Error using ==> mex at 213
Unable to complete successfully.

Error in ==> makeosmex at 25
  mex(list(i).name);

I would really appreciate it if you could help me compile the files and execute mutualinfo.m, it is my first time compiling a C source code on matlab and I have no idea how it works. I would be grateful if you could guide me step by step. Thank you for your time.

please send your answer as a comment on this page.

[HONGJING]

Actually I have exactly the same problem as Jochen Kumm cause I'm using Mac. I don't know how to fix it.

[HONGJING]

I solved my problem by recompile everything.

[Brahim HAMADICHAREF]

Although this is a fast routine, the Mutual Information value for small length vector does not seem to be very reliable when using the same vector as X and Y.
Trying the following small test code ...

clc
clear all
close all

nN = 5000;

fMIacc = zeros(1, nN);

figure
hold on
for i = 1:nN
    fVec = rand(1,i);
    fMI = mutualinfo(fVec,fVec);
    fMIacc(1, i) = fMI;
end
plot(fMIacc)
axis([-100 nN+1 0 1.5])

[Sonali]

I am getting following errors. I tried to recompile it but still getting these errors.

??? Error using ==> estpab
The two vectors/images should have the same length.
 
Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> mrmr at 50
   t(i) = mutualinfo(d(:,i), f);

[Anatoly]

Good day! thanks for interesting program. Can anyone please explain me why, having entropy(rand(10000,1))=0 (it's ok),
it gives entropy(sin(rand(10000,1)))=0.9978, BUT entropy(cos(rand(10000,1)))=0
???
:-)
I believe entropy(sin(rand(10000,1))) should also equal 0... or not? with cos(rand) or sin(rand) signals we definitely have less information than with pure rand signal?

[Anatoly]

Sorry,

I believe entropy(rand(10000,1)) should be high, cause here we have much uncertainty of values of variable.

entropy(sin(rand(10000,1))) should definitely equal entropy(cos(rand)), and, I suppose, be small, cause we have less information than with pure random signal (albeit also random)?

Can anyone clarify, please?

[Waljinder Singh]

Ok it works now for me. I will tell how it worked. Firstly download and install the compatible compilers (1. Microsoft visual 2.Microsoft Windows SDK - for windows 64 bit) according to your Matlab version. Once it is installed on your machine. Type mex -setup in Matlab. Choose the installed version and confirm by typing y. Now make sure you have all cpp extension and header files.
 Next work is to edit the cpp extension files. Open the files one by one and where you find log() type double like this: log(double()) and save. Now go in the directory where you have the m file makeosmex and run it. After this add all the files and folders to the current working directory. Run the mRMR MID/MIQ/BASE files and et voila its done !
If any body has more questions you are free to ask

[FEUP]

Hi! Thanks for this work.
The program works well, but I can't find m-file "estpab" that is use in the m-file "mutualinfo". Do you help me?
(Sorry for my english. :))

[Waljinder Singh]

Hi FEUP, As written above, follow the instructions. What OS do u use? Windows, Linux..etc.? You need to install the compatible compiler (if u don't have one)to make OS compatible mex files
  -> Also the Rows in the Data [MXN] and the Rows in the class vector [MX1] should be same.

[BUCT]

thank u

[Zeinab]

I'm working in Matlab. One file, estpab.m, is missing. Anybody knows what to do?
Thanks

[Vincent]

For everyone with error C2668 on Windows:

This can be fixed with a simple typecast of the kind:

log(2) -> log(double(2))

Just replace your source code accordingly.