Relative usually means relative to the value at the initial point x0. This means
relative violation = violation / (1 + max(confun(x0))
or some such thing, where confun means the various constraint violation functions.
You see that the constraint violation is different than the relative constraint violation. Why do optimization functions do this? In order to be more scale invariant. If you multiply your constraint functions by some scalar, the relative violation does not change.
MATLAB mathematical toolbox documentation