I have a non linear filter with unknown transmition function, which i want tio find.The data is input and output signal. Is there a builtin function in matlab for that purpose?
You can use tfestimate() in the Signal Processing Toolbox to estimate the transfer function.
In the System Identification Toolbox, there is etfe
See various model estimation functions in System Identification Toolbox such as pem, n4sid, oe, tfest, procest, ssest (the latter 3 are new in R2012a) for linear transfer function estimation. For nonlinear estimation, see nlarx, nlhw.