fmincon is a gradient-based solver, and as such does not work well with discontinuous functions. Locally, it attempts to minimize the objective function, while keeping the constraint function negative. If the constraint function jumps, well, fmincon gets confused and hunts around.
I would implement your constraint function as an appropriately shifted and scaled sinusoid, negative for the feasible points, positive for infeasible points. However, even such a smooth formulation will not necessarily give a good solution unless you give a starting point in the correct feasible interval. fmincon is a local solver, and does not look in other locations when it fails to converge to a solution starting from one point. For that, you might want to use MultiStart.
MATLAB mathematical toolbox documentation