MATLAB does not calculate the kernel parameters by itself. If there are no given values by the user, MATLAB uses default values. For the polynomial kernel the default order is 3. But you can set any polynomial order you want by calling SVMTRAIN with additional parameter.
For example with this code you create a Polynomial Kernel of order 4.
svmstruct = svmtrain(data, groups, 'Kernel_Function', 'polynomial', 'Polyorder', 4)
The RBF Kernel behaviour is similar. MATLAB uses a Kernel of this form:
k_rbf(x,y) = exp(-1/(2*sigma^2) * ||x-y||^2).
In literature this Kernel is also referred as:
k(x,y) = exp(-gamma * ||x-y||^2)
If you didn't choose any value for sigma, SVMTRAIN uses the default value of sigma = 1. For example, you can set sigma to 0.2 by call SVMTRAIN in the following way:
svmstruct = svmtrain(data, groups, 'Kernel_Function', 'rbf', 'RBF_Sigma', 0.2)
The penalty parameter C is called BoxConstraintValue in MATLAB. SVMTRAIN uses a default value of 1, but you can set this value (e.g. to 0.8) by your own by using:
svmstruct = svmtrain(data, groups, 'Kernel_Function', 'rbf', 'RBF_Sigma', 0.2, 'BoxConstraint', 0.8)
If you want to find the optimal values for (sigma,C) you should use cross validation and grid search. The idea is the following:
1. generate disjunct subsets A_i from the main set A, such that the union of the A_i equals A (see CROSSVALIND in Matlab help)
2. in a loop try parameters for (sigma,C) (so called grid search)
3. generate for each subset A_i a support vector machine svm_i with the test parameter pair (sigma,C)
4. test each svm_i with the remaining subsets A_j, j not equal to i
5. take a look on the error rate and take the pair (sigma,C) with the lowest error rate for all pairs (sigma,C) you have tested (see SVMCLASSIFY and CLASSPERF in MATLAB help)
6. calculate the final svm with the so found pair (sigma,C) based on the whole set A
You have still some parameters to choose, e.g. the number of subsets A_i and the stepsize for C and sigma. These parameters depends on your given data set and problem, so it is up to you.