As documented in the odeset function documentation page the function you specify as your custom OutputFcn must have a certain syntax and must accept certain types of calls.
"If you write a custom output function, then it must be of the form
status = myOutputFcn(t,y,flag)
The output function must also respond appropriately to these flags:"
You haven't provided much detail about the problem you're experiencing, but I suspect your custom OutputFcn does not conform to this requirement because it fails to accept the flag input argument.