Typically we add some inequality constraints to ensure a positive conditional variance in the GARCH(1,1) model, like constant > 0, arch > 0, garch > 0 and arch + garch < 1.
The numeric optimization function fmincon supports inequality constraints. Since the constraints are linear, we can put them in a matrix form like A * x < b. If the parameters are stacked like [constant, garch, arch], then the constraints can be formulated as
A = [0 1 1]
b = 1
lb = [0 0 0]
Then we can run the constrained optimization using the syntax
where the empty matrix  indicates non-existing constraints of other types.
By the way, we may replace lb by an augmented A and b with more rows, but it is not as efficient as the explicit lower bounds.