Feature selection to perform classification using Multinomial Logistic Regression
6 views (last 30 days)
Hello, I need to do feature selection in a classification situation. I have two different situations normal versus pathological and the discrimination of five classes. I'm thinking to use Multinomial logistic regression implemented in stepwisefit Matlab function. After feature selection the classification is performed using SVM with RBF kernel. Do you think its a good aproach? Must data respect some constrainsts? I havent found any paper using this aproach!.
Ilya on 6 Dec 2016
You should not use a linear model for feature selection and a nonlinear model for classification on the selected features. If you have the latest MATLAB (16b), the fscnca function in the Statistics and Machine Learning Toolbox can perform simultaneous feature selection and classification with an RBF kernel. If you do not have 16b, try sequential feature selection from sequentialfs using SVM with an RBF kernel for feature selection. Sequential feature selection could prove too slow if you have many features.
More Answers (3)
Ilya on 22 Dec 2016
You describe a procedure for selecting a set of features at fixed hyperparameter values. You do not say what you do, if anything, to optimize the hyperparameter values.
As long as you keep an independent test set not used for any sort of optimization, you are going to get an unbiased estimate of the classifier accuracy on that set. There is no overfitting in that sense.
You should keep in mind though that the 10-fold estimate of the model accuracy for the optimal feature subset is going to be biased high because the 10-fold estimate is what is being used to perform feature selection.
Some overfitting is inavoidable since sequential feature selection is greedy by nature. For example, if you start with an empty set and keep adding features as long as accuracy goes up, you are most likely going to end up with more features than necessary. The increase in accuracy may not be significant, but the feature is added to the optimal set anyway.