Problem with rectifying images from a calibrated stereo-pair system

I'm trying to use the Computer Vision System Toolbox to generate a 3-d point cloud for an object relatively close to the cameras (120 cm away). I know this distance is somewhat challenging, but I need to measure millimeter-scale objects and I am limited by the ceiling height of the laboratory (overhead cameras). My baseline is also limited to 15cm by the camera bodies. I am using two Canon T3i 18MP DSLR cameras with 18-55mm kit lenses and have run several stereo camera calibrations varying the focal length (18-55mm), distance to object (75cm-120cm), baseline (15-50cm), output image size (150kB-10MB) and checkerboard size (11.5-54mm squares). The cameras are angled inward slightly to converge on a shared central point.

During calibration I follow the guidelines recommended in the stereoCalib tutorial: at least 20 image pairs, covering most of the FOV, varying angles (< 45deg), ensuring manual focus is constant, and good lighting. The mean reprojection errors vary from 0.17-0.67 pixels but as a rule of thumb, I aim for 0.5 pixel or less.

The problem I am having is that upon rectification, the two images (red and cyan) of the object of interest appear spaced apart. This results in a stereo anaglyph with two objects of interest side-by-side rather than overlapping.

This problem persists regardless of camera setup (distance, baseline, etc.). I’m not sure if I need to change a parameter in my physical camera setup or if there is something I am missing in the script I am using. Below is the script I used to make the stereo anaglyph.

 load('stereoParams.mat')

 I1 = imread('left_camera.png'); %left
 I2 = imread('right_camera.png'); %right

 [J1, J2] = rectifyStereoImages(I1, I2, stereoParams); %valid only (default)

 imtool(stereoAnaglyph(J1, J2));

I have attached the calibration session, stereoParams object, and the two images I used to make the stereo anaglyph if that helps.

I'm using Matlab version R2014b under the University of Massachusetts Amherst’s Total Academic Headcount (TAH) license (and also a personal student account) for a PI who has purchased the Computer Vision System Toolbox for the UMass Amherst Geomechanics research group.

Thank you in advance for any help!

Kevin

Answer 1

[Dima Lisin]

Hi Kevin,

I don't think you have a problem here. A disparity of ~80 pixels for an object 120cm away from the cameras is very reasonable. In fact, if you look at your anaglyph through red-cyan stereo glasses, you will see the 3D effect.

Try computing the disparity map using the disparity function, and see how that looks. Just be sure to increase the DisparityRange, which is [0, 64] pixels by default.

[Kevin Toeneboehn]

Hi Dima,

Thank you again for your help earlier this week! Now that I’ve made an accurate scene reconstruction, I’ve started to work on getting the precision down to mm depth and I’ve run into a problem. When I minimize the baseline by having the cameras touching I have no trouble generating a valid rectification, however this certainly doesn’t yield the best depth resolution. When I move the cameras apart to get better depth information the largest valid rectangle doesn’t share any of the objects of interest. For example:

Here’s the full output view when the cameras are touching.

And here’s the largest valid rectangle after rectification.

This is great and yields a fine disparity map and scene reconstruction. However, I’d like to improve on its fine-scale depth resolution (I can already resolve pretty well to 1cm!).

Here’s what happens when I move the cameras apart.

Full output view again.

And largest valid rectangle again.

I’ve been looking at the rectifyStereoImages function and I was wondering if there is something I could do to allow for a wider baseline. Is there something I could adjust in the function or am I just limited to a very small baseline (15 cm) because of the relatively short distance to my object of interest (78-150 cm)?

Thanks for your help!

Kevin

Also, thanks for the suggestion about vision.PointTracker! I think that it will be very useful for what we are doing especially when combined with the topography from the dense reconstruction!

[Dima Lisin]

Hi Kevin,

You can try setting the OutputView to 'Full' and computing the disparity on the resulting images. The problem is that as you increase the baseline, the disparity also increases, and becomes harder to compute. Essentially, you are correct: the distance to the object is too short here. Increasing the baseline helps to measure objects further away, but it makes it hard to measure things that are too close.

There are several things you can try to increase accuracy:

• Get higher resolution cameras
• Measure the size of the checkerboard square very precisely (e. g. with a caliper). If you can measure it up to 0.1 mm, that can make a difference.
• Be sure not to use lossy compression on your images. Jpeg artifacts can affect the checkerboard corner detection accuracy. Ideally, set your cameras to acquire and store uncompressed images.
• Try different calibration settings. Try estimating tangential distortion and skew.
• If you have calibration images that result in a high reprojection error, try excluding them and re-calibrating.

Answer 2

[Alois Sing]

Hi, i have the same problem. I have already asked this question here: https://de.mathworks.com/matlabcentral/answers/231808-problem-with-rectify-stereoimages

I cant move the cameras closer together or the object farther away, because i want a low distance error. I have 54 pictures of calibration pattern and the mean error is 0.1 pixel. The focal length is with around 150mm really high, because of the low distance error. It seems, that the problem is the function rectifyStereoImages.

If the outputview is "full" it shows the hole picture and the rectification is still good, but there is a big black space on each picture. calculating the disparity from this pictures is still hard and the result is really bad. On the picture below you can see the rectified pictures with horizontal lines witch shows, that the rectification process is still working.

and this is the best result from the disparity and the reconstructScene. The depth is too far away and a lot of points are on a plane.

If the outputview is "valid" it just shows the region, wich overlaps at infinity. You can see that on the left in the next picture, on the right is the "full" picture with the surroundet region wich is the output of "valid".

Can anybody help me with this problem ? Or has a hint for calculating the disparity from the "full" pictures.

[Walter Roberson]

eric marschall comments

Not solved yet!