The error message “Non-zero degree P requires a non-zero degree Q” basically says that the conditional volatility is not well-defined in the absence of Q, because the conditional volatility is not a function of past observations when Q=0. A model specification like GARCH(1,0) or GARCH(2,0) will trigger that error.
If the data do not have any sign of volatility clustering, estimating a GARCH(1,1) may result in the estimated coefficient Q equal to zero, hence the error message. For example, estimate(Mdl,rand(100,1)) is likely to have that error message.
There is a work-around: set the optimization algorithm as interior-point so that Q will not be exactly zero.
options = optimoptions(@fmincon,'Algorithm','interior-point');
Mdl = garch(1,1);
In that case, Q can be extremely small if there is no volatility clustering found in the data, but the error message will not pop up. The trick can be useful if we want to estimate many garch(1,1) models in a FOR loop.
Hope that helps,