Thank you for reporting this odd behavior, Matt. I investigated and found that an internal check for stopping in the nonlinear least-squares code is
if iterationNumber > maxiter
Therefore, when you set 5 as the iteration limit, the check is triggered at the sixth iteration, not the fifth.
In contrast, several other solvers have the more natural
if iterationNumber >= maxiter
This stops the solver at iteration 5. So this is at least an inconsistency, if not a bug. I will now report this in our internal development system.
By the way, the reason that your test sometimes found only 5 iterations, not 6, is that the problem was solved, not that the iteration limit was hit. Your test(99) is that case.
As for why an extra 100 function evaluations are done, your code line
causes those evaluations, not the lsqcurvefit code.
Thanks again for the report,
MATLAB mathematical toolbox documentation