LQR always returns a stabilizing feedback gain.
Are there 1 or 2 eigen values that are always show up positive?
You most likely have an uncontrollable mode in your system. As Azzi mentioned you have to just try different weights, choosing Q and R is part art, part science.
If you can provide the state space (A,B,C,D) for your plant, it would be useful.