is there a way to stop an external routine called from matlab ? E.g when the routine gets stuck or takes too long (bad starting values or other reasons).
It depends on how the routine was started.
If it was started through a direct call to the program (that is, the program has a mex interface), then NO, there is no way for that MATLAB session to stop it... at least not without having launched a watchdog program before-hand.)
If it was started through a call to system() or unix() or dos() and the command line does not include the character "&" then NO, same as above.
If it was started through a call to system() or unix() or dos() and the command line contains the character "&" in the right location, then the Fortran routine would run independently of the MATLAB session. In such a case, the MATLAB program could monitor the progress and use system() to call kill() [Linux or OS-X] or taskkill (MS Windows) to shot it.