FIRQUEST searches for a quantized filter with optimal stopband rejection, using an unquantized filter as a starting point. Trial filters are generated using the error spectral shaping technique, which is described in:
J.J. Nielsen, 'Design of Linear-Phase Direct-Form FIR Digital Filters
with Quantized Coefficients Using Error Spectrum Shaping Techniques',
IEEE Transactions on Acoustics, Speech and Signal Processing,
vol. ASSP-37, pp. 1020-1026, july 1989.
Compared to ordinary rounding, the generated filters will have lower errors in the stopband, and larger errors in the passband. With the appropriate shaping filter, the stopband can be lowered by as much as 28 dB compared to ordinary rounding.