File Exchange

image thumbnail

Mutual Information computation

version 1.0 (297 KB) by

A self-contained package for computing mutual information, joint/conditional probability, entropy

4.36735
51 Ratings

301 Downloads

Updated

No License

A self-contained, cross-platform, package for computing mutual information, joint/conditional probability, entropy, and more. This package has also been used for general machine learning and data mining purposes such as feature selection, Bayesian network construction, signal processing, etc.

Another related package for minimal redundancy feature selection is also available at the Matlab Central exchange site, under the category of "Biotech and Pharmaceutical".

A simple demo is called demo_mi.m.

*** Please note that downloading or use of this package means acceptance of the license of this package. In short, this package is free to non-profit use but cannot be re-distributed in any form, including revised forms, without the explicit permission for the author, Hanchuan Peng. See readme file for further information. ***

Comments and Ratings (95)

xiaobo zhang

imu931

imu931 (view profile)

@Erdem Isenkul proposed a practical and useful way. I just would like to add one point, that we should search ' /= log(2)' instead of 'muInf /= log(2)', since in some .cpp files there are 'entropy /= log(2)'.

Best

I tried executing the mrmr_miq_d.m file. It says that it does not have enough input parameters. I need help resolving this

Ok, I have transformed all my variables into uint64. Now I get the error:

>> z = mutInfo(x, y)
Undefined function 'sparse' for input arguments of type 'uint64'.

Error in mutInfo (line 19)
Mx = sparse(idx,x,1,n,k,n);

Any held?

My values are not integers. How can I use your function? I want to explain the dependence between acceleration and emissions. Since I have acceleration, I have negative values too.

Erdem Isenkul

If you get an error using the functions here is the solution;
First of all you need to "mex" all of .cpp files in MATLAB.
If you get an error about double,long double and etc. Open that cpp file in notepad and find " muInf /= log(2); " line.

Change this line with " muInf /= (double)log((double)2); "

After edit save cpp file and mex again.

You need to do this step for every cpp file which you got errors.

Regards.

Helix Hsu

huang hai

What kind of probability density estimation is used in estpab

Mbvalentin

Solving the problem that Woodrow mentioned is usually easy: simply multiply your array by a constant and round. i.e, if you're working with normalized grayscale images (which range from 0 to 1), simply multiply them by (2^nbits-1) and round, where nbits is the number of bits that variable has, for instance, 8 for uint8, 16 for uint16, etc.

So it would look something like:

x = round((2^8-1)*rand(100,1));
y = x;
mutualinfo(x,y);

Felix Chavez

Error, demo don't work

mutualinfo(a,b)
Undefined function 'estpab' for input arguments of type 'double'.

Error in mutualinfo (line 21)
[p12, p1, p2] = estpab(vec1,vec2);

Error in demo_mi (line 25)
mutualinfo(a,b)

Woodrow

WARNING!!! WARNING!!!! WARNING!!!

This code assumes that you are working with discrete (integer) variables. If you are not, the code rounds your variables, turning them into integers without warning you. This will cause major errors if, for example, your variables are not integers and are less than 0.5. It will cause less extreme errors, but errors nonetheless, if your variables are larger, but not integers.

For example the following code returns mutual information of zero! Which is obviously false.

x=[0 1 1 1 0 0 1 0 1 0 1 1 1 1 0 0 0 1 1]*0.1;
y=x;
mutualinfo(x,y)

hello,when i solve the problem "Undefined function 'estpa' for input arguments of type 'double'",then i meet the new problem"Error using estpab
Out of memory. Type HELP MEMORY for your options." i hope anyone can help me please.

Hello
Can anybody guide me how to use mrmr?
Thank you

Whenever run demo_mi get the following error.
Undefined function or variable 'estpa'.

Error in entropy (line 34)
    [p1] = estpa(vec1);
makeosmex working fine and followed some of suggestions like convert log(2) log(2.0) while theoretically this should not be the problem.
In the directory estpa is available but without I think input parameters that we use in calculating entropy

Raviajot Kaur

I am getting following errors. I tried to recompile it but still getting these errors.

??? Error using ==> estpab
The two vectors/images should have the same length.

Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> mrmr at 50
t(i) = mutualinfo(d(:,i), f);

mutualinfo returns zero value always may i know what's the reason or my coding may be wrong

ashish jadhav

Cui Can

Jenifful Ai

When I run the demo code ,I find the same problems mentioned below.My platform is Matlab2015 in Windows x64.Finally, I replaced all "log(2)" notations in all the .cpp files with "log(2.0)".And then install sdk if you do not have a supported sdk ,input'mex -setup'in your command window,then follow the default choices .

Lennart

Compiled nicely after replacing log(2) with log(2.0) in all .cpp files as andre and others mention.

However when I run the code it crashes matlab. I use matlab 2013a and windows 7 64 bit. It seems to work for others though when I read the comments below.

Andre Silva

Thanks for the submission.

I found that the mex files did not work for my computer. (Windows 7 x64). So I had to use the "mex -setup" to build mex files from the .cpp source files.

To do this all I did was edit all "log(2)" notations in all the .cpp files to "log(2.0)". I appreciate this suggestion from the previous comments here.

Hope this helps someone.

Andre Silva

Kyrion

Kyrion (view profile)

Adam

Adam (view profile)

alireza

this code not run at all and nobody answer this problem!!!!thanks really

saisakul C

Hi All,

For those who can't run the demo file because of 'undefined function' error. You need to:

1) Run this command: list = dir('*.cpp');
to get the list of files.

2) For all the files in list, change log(2) to log(2.0). You can get all the files' names by:

for i=1:length(list)
list(i).name
end

2) Run 'makeosmex.m'

Now this should works like a dream.

Nawaz Pasha

I got following error, on running demo_mi.m

Undefined function 'estpab' for input arguments of type 'double'.

Error in mutualinfo (line 21)
[p12, p1, p2] = estpab(vec1,vec2);

Error in demo_mi (line 25)
mutualinfo(a,b)

What's wrong ?

Hang Zeng

I also get the problem "Undefined function 'estpa' for input arguments of type 'double'". I find the estpa is cpp file which is a C++ file. I use Win8 62 bit application and Matlab 2011b. What's wrong with it?

Wadim

Wadim (view profile)

when the range of the features vectors is large, let's say 1-10000 estpab() calculates probibilty density with the same length (10000 in this case). Now if both feature vectors have a large range, lets say the other one is 1-10000 too, then the joint pdf computed by estpab is a 10000 x 10000 matrix and the computation if MI takes very long time. Is there a way to modify the algorithm in order to avoid this problem?

Affan

Affan (view profile)

I am getting following error message when I running mi software package:
even I am running this mi package on Intel core i5 with 8 GB memory.

----------------
??? Error using ==> estpab
Out of memory. Type HELP MEMORY for your options.

Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> simple_main_MISS_AR at 174
              mi_1(i)= mutualinfo(feat_2,e_1(i,:));
--------------

feat_2 and e_1(i,:) are 1x600 vectors.

please help regarding this.

Thanks

Affan

Douglas Jones

I also got the ESTPAB problem. When I tried running makeosmex to recompile mex files I got the error below.

>>
xcodebuild: error: SDK "macosx10.7" cannot be located.
xcrun: error: unable to find utility "clang++", not a developer tool or in PATH

    mex: compile of ' "estcondentropy.cpp"' failed.

Unable to complete successfully.

Error in makeosmex (line 25)
  mex(list(i).name);
<<

None of the solutions on this board worked for me (e.g. changing log(2) to log(2.0) or to log(double(2)), etc.

The problem turned out to be that I have Xcode 5.0 and Mac OS10.8, which apparently does not work out-of-the-box. The solution suggested by the Mathworks support team worked for me:
http://www.mathworks.com/matlabcentral/answers/103904

After that, makeosmex compiled for me and I was able to run demo_mi.

Other than that this package looks good.

Anamika

I am getting a problem "Undefined function 'estpa' for input arguments of type 'double'". I use Win7 32 bit application. Actualy what is the problem?

Yongning

Is there any assumption made when calculating mutualinfo(x, y) if one of vector x,y is binary?
In my data, y is binary and I tried several different x. Surprisingly, several different x give the same mutualinfo to the last digit of double type.
I think some assumption must be made on estimating kernel density. Could author give some insight?

Lei Yang

good

Pedro Silva

btw, I'm running Matlab 64-bit with linux, which process limit is 8 TB

Davis

Davis (view profile)

Maybe I'm missing something, but I think this code produces weird behavior with vectors of low values, e.g. returning entropy of 0 for [1:10] * .0001.

Run this code and see for yourself:
a = [];
for i = 1:20e4
a(i) = entropy([1:10] .* i/10e4);
end
plot(a)
set(gca,'xticklabel',[0:.2:2])
ylabel('Est. Entropy')
xlabel('Scaling factor')

Jing Wang

Thanks you all. For win 7 64-bit version, the contents of all .cpp files should be changed from log(2) to log(2.0) if log(2) is used.

Gordon

Gordon (view profile)

ESTPAB problem.

The mex dlls compiled I believe are 32bit. If you have a 64bit MATLAB application you will not be able to run this software.

To solve this

1) Run makeosmex.m found in the source directory "mi".

2) This failed for me as the log functions in the c++ files needed to be adjusted.

3) For each failure, find the line in the .cpp file where the mex compiler falls over (this will be output to the matlab command window) and replaced log(2) with log(2.0)

Best of luck.

what is condvec in the condmutualinfo.m file???Can anybody please help me?

yawai tint

I have some problem for using this code.When I run the program, I have the error "Invalid mex-file: the ..........estentropy.dll file isn't a valid win64 application".
Could somebody explain to me what is the problem?
I am using the matlab version (R2010) on a windows machine.
Thank you very much!

khalil ben

May

May (view profile)

Thank you

Stefan

Stefan (view profile)

function 'mutualinfo' can not find function 'estpab'

where is the problem?

Onur Ugurlu

I wanted to try say,
how can I change the time delay of mutual function ??

THanks...

Onur Ugurlu

Hi everyone,
How can ı add decay function or if it already in the program, how can ı can chanhe its valuse.?
if someone can help me, he will save my life :)
Thanks..

Mohammed Ambu

Thank you

Ryan

Ryan (view profile)

Does anyone have a recommendation on how to compute entropy conditioned on multiple variables e.g. H(X|Y,Z,W)?

Thanks!

Wei Xiong

I want to say thank you so much. You saved my day.

Vincent

For everyone with error C2668 on Windows:

This can be fixed with a simple typecast of the kind:

log(2) -> log(double(2))

Just replace your source code accordingly.

Zeinab

Zeinab (view profile)

I'm working in Matlab. One file, estpab.m, is missing. Anybody knows what to do?
Thanks

BUCT

BUCT (view profile)

thank u

Hi FEUP, As written above, follow the instructions. What OS do u use? Windows, Linux..etc.? You need to install the compatible compiler (if u don't have one)to make OS compatible mex files
  -> Also the Rows in the Data [MXN] and the Rows in the class vector [MX1] should be same.

FEUP

FEUP (view profile)

Hi! Thanks for this work.
The program works well, but I can't find m-file "estpab" that is use in the m-file "mutualinfo". Do you help me?
(Sorry for my english. :))

Ok it works now for me. I will tell how it worked. Firstly download and install the compatible compilers (1. Microsoft visual 2.Microsoft Windows SDK - for windows 64 bit) according to your Matlab version. Once it is installed on your machine. Type mex -setup in Matlab. Choose the installed version and confirm by typing y. Now make sure you have all cpp extension and header files.
 Next work is to edit the cpp extension files. Open the files one by one and where you find log() type double like this: log(double()) and save. Now go in the directory where you have the m file makeosmex and run it. After this add all the files and folders to the current working directory. Run the mRMR MID/MIQ/BASE files and et voila its done !
If any body has more questions you are free to ask

Anatoly

Sorry,

I believe entropy(rand(10000,1)) should be high, cause here we have much uncertainty of values of variable.

entropy(sin(rand(10000,1))) should definitely equal entropy(cos(rand)), and, I suppose, be small, cause we have less information than with pure random signal (albeit also random)?

Can anyone clarify, please?

Anatoly

Good day! thanks for interesting program. Can anyone please explain me why, having entropy(rand(10000,1))=0 (it's ok),
it gives entropy(sin(rand(10000,1)))=0.9978, BUT entropy(cos(rand(10000,1)))=0
???
:-)
I believe entropy(sin(rand(10000,1))) should also equal 0... or not? with cos(rand) or sin(rand) signals we definitely have less information than with pure rand signal?

Sonali

Sonali (view profile)

I am getting following errors. I tried to recompile it but still getting these errors.

??? Error using ==> estpab
The two vectors/images should have the same length.
 
Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> mrmr at 50
   t(i) = mutualinfo(d(:,i), f);

Although this is a fast routine, the Mutual Information value for small length vector does not seem to be very reliable when using the same vector as X and Y.
Trying the following small test code ...

clc
clear all
close all

nN = 5000;

fMIacc = zeros(1, nN);

figure
hold on
for i = 1:nN
    fVec = rand(1,i);
    fMI = mutualinfo(fVec,fVec);
    fMIacc(1, i) = fMI;
end
plot(fMIacc)
axis([-100 nN+1 0 1.5])

HONGJING

I solved my problem by recompile everything.

HONGJING

Actually I have exactly the same problem as Jochen Kumm cause I'm using Mac. I don't know how to fix it.

Jochen Kumm

I am running matlab on mac osx 10.4 (matlab 7.7.0 R2008b) and also on the unix server (matlab 7.11.0 R2010b), would you please tell me how I can compile the files on both of these machines.

I could run makeosmex.m on the unix machine without any error but when I tried to run demo_mi.m, it gave me the following error (I actually replaced log(2) with log(2.0) in estmutualinfo.cpp as well):

??? Undefined function or method 'estpab' for input arguments of
type 'double'.

Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

Error in ==> demo_mi at 25
mutualinfo(a,b)
 

On mac osx I even could not run makeosmex.m, it gave me the following error:

building mex(dll) of estcondentropy.cpp
  mex(list(i).name);
/Applications/MATLAB_R2008b.app/bin/mex: line 1026: gcc-4.0: command not found
/Applications/MATLAB_R2008b.app/bin/mex: line 1005: gcc-4.0: command not found
/Applications/MATLAB_R2008b.app/bin/mex: line 1: g++-4.0: command not found

    mex: compile of ' "estcondentropy.cpp"' failed.

??? Error using ==> mex at 213
Unable to complete successfully.

Error in ==> makeosmex at 25
  mex(list(i).name);

I would really appreciate it if you could help me compile the files and execute mutualinfo.m, it is my first time compiling a C source code on matlab and I have no idea how it works. I would be grateful if you could guide me step by step. Thank you for your time.

please send your answer as a comment on this page.

Hello, I do not understand the parameter d and f clearly. What will be the value of d? is it individual band for a hyperspectral image, and what will f then? Please inform me.Its better if one can send me with some example data set.

Thanks
Ali

Hello, I o not understand the parameter d and f clearly. What will be the value of d? is it individual band for a hyperspectral image, and what will f then? Please inform me.

Thanks
Ali

Ali Farzan

I have the same problem as Benjamin Mack.
How can I fix it?

Tried mutual information computation code. Works well!

Jong Rah

Nanye

Nanye (view profile)

I was using mutualinfo() and found the following errors

pure virtual method called
terminate called without an active exception
mutualinfo:778(<D,M> <D,C>):(46912990672832, 2433) -> (<T,T>)

 Any hint? Thanks!

Mihály

Hi,
I re-compiled the cpp files with mex to match te current lib versions, but when I compute a mutual information, I always get 0. Did anybody encounter this problem? (I use Linux)

Thanks.

wang

wang (view profile)

Hi, Marie, have you compiled the estentropy.cpp on your computer?

Marie

Marie (view profile)

When I run the program, I have the error "Invalid mex-file: the ..........estentropy.dll file isn't a valid win32 application".
Could somebody explain to me what is the problem?
I am using the matlab version (R2009) on a windows machine.
Thank you very much!

wang

wang (view profile)

I find the problem.
define "pabhei" "pabwid" with "long" is ok.
about line 32-34 in "estcondentropy.cpp" .

Is it right?

wang

wang (view profile)

Hi, Prof Peng,

I still met one problem when I compile "estcondentropy.cpp" by run "mex estcondentropy.cpp" after I changed log(2) to log(2.0).

The other files can be compiled successfully. Can you help me for compiling "estcondentropy.cpp"?

Thank you!
Simbest Wang

thank u for the toolbox...

Omid Aghazadeh

I'm sure the log errors are solved by now. It is obvious that you need to add a .0 to the log(2) to solve the problem if you get that error. (change log(2) to log(2.0)).

Benjamin Mack

Hello,
I try to use this programm but I am totally new to matlab (but have experience in R). When I call the function
mrmr_mid_d (variables, response, 25)
the following error occurs:
??? Undefined function or method 'mrmr_mid_d' for input arguments of type 'double'.
The same occures when I convert my data in int8 or single.

something similar happens when I try to use the demo_mi.m :

mutualinfo(a,b)
??? Undefined function or method 'estpab' for input arguments of type 'double'.
Error in ==> mutualinfo at 21
[p12, p1, p2] = estpab(vec1,vec2);

I think my problem is very a very basic one because I do not know anything about Matlab.
Could anybody please help me. It is very important for me to use this programm and the online version does not work with my data set because it is too large.

Thank you very much in advance,
Ben
 

Raphael

I am trying to compile mex files on Unix with Matlab 7.7.0(2008b) using makeosmex.m. I get the following error messages:
building mex(dll) of estcondentropy.cpp
estcondentropy.cpp:19:23: error: miinclude.h: No such file or directory
estcondentropy.cpp:21: error: ?mxArray? has not been declared
estcondentropy.cpp:21: error: expected ?,? or ?...? before ?*? token
estcondentropy.cpp:21: error: ISO C++ forbids declaration of ?mxArray? with no type
estcondentropy.cpp: In function ?void mexFunction(int, int**, int, int)?:
estcondentropy.cpp:24: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:26: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:32: error: ?prhs? was not declared in this scope
estcondentropy.cpp:32: error: ?mxGetPr? was not declared in this scope
estcondentropy.cpp:33: error: ?mxGetM? was not declared in this scope
estcondentropy.cpp:34: error: ?mxGetN? was not declared in this scope
estcondentropy.cpp:41: error: ?mexErrMsgTxt? was not declared in this scope
estcondentropy.cpp:59: error: ?log? was not declared in this scope
estcondentropy.cpp:65: error: ?log? was not declared in this scope
estcondentropy.cpp:67: error: ?mxREAL? was not declared in this scope
estcondentropy.cpp:67: error: ?mxCreateDoubleMatrix? was not declared in this scope

Suggestions?!

Carlos G-M

I can't compiler the file estmutualinfo.cpp.
estmutualinfo.cpp
estmutualinfo.cpp(42) : warning C4267: 'initializing' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(43) : warning C4267: 'initializing' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(55) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(58) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(65) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(67) : warning C4267: '=' : conversion from 'size_t' to 'long', possible loss of data
estmutualinfo.cpp(120) : error C2668: 'log' : ambiguous call to overloaded function
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(567): could be 'long double log(long double)'
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(519): or 'float log(float)'
        C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\math.h(121): or 'double log(double)'
        while trying to match the argument list '(int)'

Hanchuan

Quite some people ask for mex files for different operating system versions, such as the latest Windows, or Mac OS X 10.6 Snow Leopard, etc. Please note that the mex functions can simply be re-generated by running the "makeosmex" command when you enter the folder of this toolbox (I assume you know how to set up mex compiling environment). For Windows, I use mingw to compile.

Bahaa Khalil

Many thanks for your contribution, is it possible to adjust estpob file to be able to calculate mutual information for continuous data

Yan Jin

Thank you for your source code! It's very good^^

ming ma

very good

Guenther Eibl

I just needed the normalized MI which was easy to find and use

Usha Nair

i want to normalise the mutualinfo to the range (0-1) for comparison of different time series. Could you please help me by proper suggestion for this. Is it possible to change the number of bins used for the histogram so that all the data sets can be analyzed using the same scale.

Dedy H

Can you help me to find joint probability for multivariate data....???how can I contact with you???

sumit maheshwari

i want to write joint probabilty as a function or want to express as an equation....
please help me.......

Fernando Diaz

I need un function that is needed in mutualinfo (estpab)

Mike B

Rui Liu

very good!!

Peter Chow

a nice piece of work, make my life much easier!

ali shrof

Xiang Y

great! easy to use.

MATLAB Release
MATLAB 7 (R14)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video