First, based on what you wrote, I would guess that you may be confusing camera 1 and camera 2. According to your stereoParams object, your cameras are 9.1 cm apart. So if you are measuring locations of objects relative to camera 2 instead of relative to camera 1 that would explain most of the error in X.
Second, please keep in mind that the world coordinate system (assuming that you are using disparity() and reconstructScene() has its origin at the principal point of camera 1, which is located somewhere inside the camera casing. So that becomes hard to measure. A better test of accuracy of the reconstruction would be to measure a distance between two points in the world.
So, please check that you don't have camera 1 and camera 2 mixed up. If that does not help, could you please post a sample pair of images? Also, please clarify how exactly you are doing the reconstruction. Are you calling rectifyStereoImages followed by disparity followed by reconstructScene? If so, then of course you should be using the rectified images to look up the 3D world coordinates.