Perhaps this is very simple question. I have been trying to find it. How can we determine the order of ARMA and AR coefficient in Matlab? What I mean here is not merely trial and error.
Any suggestion is appreciated
One way would be to state-space estimation which lets you inspect the Hankel singular values for model order:
% pick optimal order in the 1:10 range: model = n4sid(data, 1:10);
For more information, see: http://www.mathworks.com/help/ident/ug/identifying-state-space-models.html
Model order selection is often a tricky business. Inspecting the Hankel singular values is one many methods. There are many such methods, arguably one as good or worse as the other.
Specifically in the case AR/ARMA models, you may look into the model residues. Specifically, if you have used a "sufficient" model orders, the residuals should look white. Note that the residuals may look white for many model order selections. There are statistical hypothesis testing procedures (e.g. Chi-2 test, etc.) for both sample auto-correlation of residuals and cross-correlation of residuals and input; Ljung's book has a lot of details on all of this.
Hope that helps.