Feature Selection Based on Interaction Information

Self-contained package for feature selection based on mutual information/interaction information.

Description

This is a self-contained package for running feature selection filters: Given a (usually large) number of noisy and partly redundant variables and a target choose a small but indicative subset as input to a classification or regression technique.
For background information, see e.g: Gavin Brown, 'A New Perspective for Information Theoretic Feature Selection', Artificial Intelligence and Statistics, 2009.

The Matlab function select_features.m includes several previously published methods as special cases, such as FOU, MRMR, MIFS-U, JMI, and CMIM. It allows for higher-order interaction terms, forward and backward search, priors, several redundancy weighting options, and pessimistic estimates.
 
Auxiliary functions for discretization, construction and marginalization of probability tables, conditional entropy, mutual information and interaction information are included and are usable by themselves. See demo_feature_select.m for examples.

• MATLAB release: MATLAB 7.8 (R2009a)
• Other requirements: Some options of quantize.m rely on the statistics toolbox.

[Hopester Hope]

Perhaps I'm being dense, but I'm getting errors whenever I use these functions. For example, the script:

X=rand(10,1);XQ=quantize(X,'levels',2);[X XQ]

throws the error:

'??? Undefined function or method 'quantile' for input arguments of type 'double'.

The script comes directly from your help page, so I guess it should work... Any ideas?

[Stefan Schroedl]

the quantile function is defined in the statistics toolbox, maybe you don't have it installed?

[Hopester Hope]

I see you're right. I was actually hoping to build something that didn't depend on the statistics toolbox, so I guess this isn't it. Thanks for keeping the code itself quite clear though; it's a good place for me to start to build what I need.

[asma]

The algorithms are so slow, is there is a way to improve the speed?

i have a feature space of 72 columns and i need to reduce them into 20 or 15.

[june]

function [varargout] = select_features(features, target, max_iter, varargin)
what target stand for?feature?or class?

[Alexander Schäfer]

Hey a great script!! But sometimes I get this error:

??? Error using ==> marginal at 37
length of dimension vector incompatible with size of probability table

Error in ==> select_features>update_comb at 491
        p_xy = marginal(p_zxy, ~ofn(1, deg+1), 1); % all variables except target

Error in ==> select_features>update_degree at 457
            use_full = update_comb(var_update, var_sel, var_sgn, [], deg);

Error in ==> select_features at 382
            needs_full = update_degree(i, sel, sgn, flag_sel, d);

Error in ==> predict_club2 at 96
             test=select_features(traindata, trainlabels,feature_nr);