How to make two matrices be mutually orthogonal?

If there are two matrices A, B.

There mutual orthogonality can be given as inner product of B^T and A.

(Different with orthogonal of one matrix, s.t (A^T)·A = I )

I want to make A & B more mutually orthogonal, it means decreasing of (B^T)·A.

Then, I have to multiply some matrix Q to both.

Finally, the question is which way is possible to find Q.

such that (QB)^T·QA < (B^T)·A .

(I guess it is need to use some optimization theory.)