Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
SVM hyperparameter selection

Subject: SVM hyperparameter selection

From: Barry Greene

Date: 25 Jul, 2012 13:55:13

Message: 1 of 6

Hi all,

I am trying to do a fast feature election and hyperparameter selection for an SVM. I have tried a grid search approach but found it to be far too slow.
I am trying instead to use nested function handles to do this based on the example given using fminsearch (http://www.mathworks.co.uk/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16) but can't get it to work with code snippet below:


c = cvpartition(labels,'k',10);
fun2 = @(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));
fun3 = @(X,y)sequentialfun(fun2,X,y);
minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);
opts = optimset('TolX',5e-4,'TolFun',5e-4);
[searchmin fval] = fminsearch(minfn,randn(2,1),opts)
 

Subject: SVM hyperparameter selection

From: Alan_Weiss

Date: 25 Jul, 2012 17:49:15

Message: 2 of 6

On 7/25/2012 9:55 AM, Barry Greene wrote:
> Hi all,
>
> I am trying to do a fast feature election and hyperparameter selection
> for an SVM. I have tried a grid search approach but found it to be far
> too slow.
> I am trying instead to use nested function handles to do this based on
> the example given using fminsearch
> (http://www.mathworks.co.uk/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16)
> but can't get it to work with code snippet below:
>
>
> c = cvpartition(labels,'k',10);
> fun2 =
> @(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));
> fun3 = @(X,y)sequentialfun(fun2,X,y);
> minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);
> opts = optimset('TolX',5e-4,'TolFun',5e-4);
> [searchmin fval] = fminsearch(minfn,randn(2,1),opts)
>

Did you see the documentation example on fitting a SVM using cross
validation?
http://www.mathworks.com/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16

Alan Weiss
MATLAB mathematical toolbox documentation

Subject: SVM hyperparameter selection

From: Alan_Weiss

Date: 25 Jul, 2012 17:59:31

Message: 3 of 6

On 7/25/2012 1:49 PM, Alan_Weiss wrote:
> On 7/25/2012 9:55 AM, Barry Greene wrote:
>> Hi all,
>>
>> I am trying to do a fast feature election and hyperparameter
>> selection for an SVM. I have tried a grid search approach but found
>> it to be far too slow.
>> I am trying instead to use nested function handles to do this based
>> on the example given using fminsearch
>> (http://www.mathworks.co.uk/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16)
>> but can't get it to work with code snippet below:
>>
>>
>> c = cvpartition(labels,'k',10);
>> fun2 =
>> @(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));
>> fun3 = @(X,y)sequentialfun(fun2,X,y);
>> minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);
>> opts = optimset('TolX',5e-4,'TolFun',5e-4);
>> [searchmin fval] = fminsearch(minfn,randn(2,1),opts)
>>
>
> Did you see the documentation example on fitting a SVM using cross
> validation?
> http://www.mathworks.com/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16
>
>
> Alan Weiss
> MATLAB mathematical toolbox documentation

Sorry, I now see that your question is exactly about this example. Can
you please describe what you mean when you say you "can't get it to
work?" Is there a syntax error? Is it too slow? Does it give a poor
solution? And, by any chance, do you have access to Global Optimization
Toolbox? In my experience, patternsearch works well on this type of problem.

Alan Weiss
MATLAB mathematical toolbox documentation

Subject: SVM hyperparameter selection

From: Ilya Narsky

Date: 25 Jul, 2012 19:34:35

Message: 4 of 6

"Barry Greene" <barrygreene20@gmail.com> wrote in message
news:juots1$7n$1@newscl01ah.mathworks.com...
> Hi all,
>
> I am trying to do a fast feature election and hyperparameter selection for
> an SVM. I have tried a grid search approach but found it to be far too
> slow.
> I am trying instead to use nested function handles to do this based on the
> example given using fminsearch
> (http://www.mathworks.co.uk/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16)
> but can't get it to work with code snippet below:
>
>
> c = cvpartition(labels,'k',10);
> fun2 =
> @(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));
> fun3 = @(X,y)sequentialfun(fun2,X,y);
> minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);
> opts = optimset('TolX',5e-4,'TolFun',5e-4);
> [searchmin fval] = fminsearch(minfn,randn(2,1),opts)
>

You seem to misunderstand what variable z does in your code. Your

fun2 =
@(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));

takes array z that exists in your workspace *when you define fun2* and uses
these values as constant parameters. If you change z later, this has no
effect on the behavior of fun2.

This definition

minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);

makes no sense to me because z is not passed to crossval. That is, minfn
always returns the same output.

If you are interested in a feature selection method that works, as opposed
to a feature selection that is novel and can be published, I would recommend
backward elimination provided by function sequentialfs. It can be slow for
many features. If you have Parallel Computing Tlbx, you can speed it up by
setting 'UseParallel' to 'always' (see the doc for 'options' passed to
sequentialfs).

If you work with linear SVM, you can judge variable importance from the
coefficient magnitude. In that case, you can skip the step where you
evaluate the predictive performance of a reduced model by cross-validation
or using an independent test set. You would still run backward elimination,
but with a quicker turnaround time. Here is a paper on that: Machine
Learning, 46, 389422, 2002, Gene Selection for Cancer Classification using
Support Vector Machines by Guyon et al. The datasets in this paper have
thousands of genes (variables).

-Ilya
 

Subject: SVM hyperparameter selection

From: Barry Greene

Date: 25 Jul, 2012 19:39:23

Message: 5 of 6

Thank you.
Unfortunately I don't have the optimization toolbox, only: statistics, signal processing and bioinformatics.
In the code snippet I have provided I am trying to use sequentialfs and fminsearch to do feature selection and hyperparameter selection for an SVM.
I have tried a number of variants on the code snippet provided and I get a variety of syntax errors related to the number of the output variables produced by the function handles. There are three layers of nested function handles included; for sequentialfs, fminsearch and crossval.
Is it possible to do what I am asking using function handles? i.e. I am looking for a fast method to implement nested cross-validation, where feature and model selection are done within the innermost loop of cross validation.

Thanks,
_Barry

Alan_Weiss <aweiss@mathworks.com> wrote in message <jupc63$2nq$1@newscl01ah.mathworks.com>...
> On 7/25/2012 1:49 PM, Alan_Weiss wrote:
> > On 7/25/2012 9:55 AM, Barry Greene wrote:
> >> Hi all,
> >>
> >> I am trying to do a fast feature election and hyperparameter
> >> selection for an SVM. I have tried a grid search approach but found
> >> it to be far too slow.
> >> I am trying instead to use nested function handles to do this based
> >> on the example given using fminsearch
> >> (http://www.mathworks.co.uk/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16)
> >> but can't get it to work with code snippet below:
> >>
> >>
> >> c = cvpartition(labels,'k',10);
> >> fun2 =
> >> @(xtrain,ytrain,xtest)crossfun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2)));
> >> fun3 = @(X,y)sequentialfun(fun2,X,y);
> >> minfn = @(z)crossval('mcr',Data,labels,'Predfun',fun3,'partition',c);
> >> opts = optimset('TolX',5e-4,'TolFun',5e-4);
> >> [searchmin fval] = fminsearch(minfn,randn(2,1),opts)
> >>
> >
> > Did you see the documentation example on fitting a SVM using cross
> > validation?
> > http://www.mathworks.com/help/toolbox/bioinfo/ug/bs3tbev-1.html#bs3tbev-16
> >
> >
> > Alan Weiss
> > MATLAB mathematical toolbox documentation
>
> Sorry, I now see that your question is exactly about this example. Can
> you please describe what you mean when you say you "can't get it to
> work?" Is there a syntax error? Is it too slow? Does it give a poor
> solution? And, by any chance, do you have access to Global Optimization
> Toolbox? In my experience, patternsearch works well on this type of problem.
>
> Alan Weiss
> MATLAB mathematical toolbox documentation

Subject: SVM hyperparameter selection

From: Barry Greene

Date: 25 Jul, 2012 19:49:15

Message: 6 of 6

"Basically I am looking for a fast method to implement nested cross-validation, where feature and model selection are done within the innermost loop of cross validation.
I am trying to use sequentialfs and fminsearch to do feature selection and hyperparameter selection for an SVM.
I have tried a number of variants on the code snippet provided and I get a variety of syntax errors related to the number of the output variables produced by the function handles. There are three layers of nested function handles included; for sequentialfs, fminsearch and crossval.

I have used both SVMs and sequentialfs extensively however I have not found a satisfactory (fast) way to optimize both and produce an unbiased estimate of model performance. For this reason I am trying to implement nested cross validation.

Thanks,
_Barry

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us