I apologize that I cannot answer each comment here. The two-layer model implemented in the OFT has been thoroughly checked numerous times. It has been used for a couple of years now to compare against our results in various experiments. Plus it was used to verify the three-layer model, whenever possible. No mistakes have been found in the recent time, the two-layer code is stable. Therefore I am pretty sure there are no mistakes in that part of the code. Mistakes (typos) do occur in books and papers however. Also, different notations and/or assumptions can be made, e.g. concerning the coordinate system or polarisation direction.
I think there might be a mistake.
In the file modeField.m
according to Snyder's book
m is odd, it is HE mode
But in your code,
f = @(nu,phi) cos(nu*phi);
g = @(nu,phi) -sin(nu*phi);
I tried out the demo oftDemo without modifications. It found modes and showed me the field profiles, but it also claimed that there are "traceMode WARNING: too little amount of points in the mode" and the dispersion curves that i got were empty.