This is a question of thinking deliberately. If I understand you correctly, you want to find the control variable lambda = (lambda(1),lambda(2)) that gives the lowest value of RMSD. But RMSD does not depend explicitly on lambda.
Here is where deliberate thinking comes in. fminsearch wants a function that accepts lambda and gives the value of RMSD. If you think about it, you should realize that is exactly what you want, too: the value of RMSD for a given value of lambda. Either you can compute these values or you cannot. If you can compute them, then think about how you do it, and write a MATLAB function that takes a value lambda and returns the RMSD value (this function may well call RMSD).
MATLAB mathematical toolbox documentation