Can you please post your disparity map and the stereoParameters object? Also, it would help if you could post the images.
Off the top of my head, I would try restricting the range of the Z-coordinates. In other words, exclude the points where Z is negative or too large from the point cloud. It may well be that the point cloud you are seeing is dominated by noise.
Also, check if camera 1 is on the left or on the right, as you look in the same direction as the cameras.