Hi Zeynep Kahraman
The equation itself can be approach in different ways, Symbolic or numeric.
I tend to go for numeric 1st, kind of habit.
f2= tan((pi*d*n1/lambda0)-(sin(acos(n2/n1))./(theta/(2*d)) *pi/2)-((sin(acos(n2/n1)))^2./(sin(theta)).^2-1).^.5)
That is the mathematical equation, I had to guess all the inputs, so if you supply the input parameters I will plot again with with d n1 n2 that you choose.
Regarding the type of waveguide:
The equation resembles a dielectric interface for optical wave-guide.
Literature reference [RAMO]
Fields and Waves in Communications Electronics, 2nd ed
by Simon Ramo, John Whinnery, Theodore Duzer. Ed: JWiley&Sons
Would you please give some details about the geometry of the interface the equation models? is it possible for you to attach a diagram to the question?
If you find the above ok for you to proceed in your work then it means you find this answer useful. Then would you please be so kind to consider marking my answer as Accepted Answer?
To any other reader, if you find this answer useful please consider clicking on the thumbs-up vote link
Zeynep if you need further development, perhaps you would like to consider supplying values for d n1 n2 and lambda0.
thanks in advance for time and attention