Large error in sparse 3D point cloud

I am working on sparse reconstruction using a calibrated stereo pair. This is the approach I have taken step by step (thanks to Dima):

1- I Calibrated my stereo cameras using the Stereo Camera Calibrator app.

2- I Took a pair of stereo images and Undistorted each image.

3- I Detect, extract, and match point features.

4- I Use the triangulate function to get 3D coordinates of the matched points by passing the stereoParametes object into triangulate. The resulting 3D coordinates are with respect to the optical center of camera 1 and it is in millimeters.

The problem is that the point clouds seems to be warped and curved towards the edges of the image. at first it seemed like a barrel distortion of the lenses to me. so I recalibrated the bumblebee XB3 cameras using MATLAB camera calibrator app. but this time I used 3 radial distortion coefficients and also included tangential and skew parameters. but the results are the same. I also tried Caltech's camera calibration toolbox but it had the same results as MATLAB. also another problem is that the Z values in the point cloud are all negative but I am thinking that might be coming from the fact that I am using right camera as camera 1 and left camera as camera 2 as oppose to what MATLAB's coordinate system is in the link attached.

I have attached couple of pictures of 3D point cloud from both sparse and dense 3D reconstruction. I am not interetsed in Dense 3D but just wanted to do it to see if the problem still exist which it does. I believe that means the main problem is with the images and camera calibration rather than algorithms.

Now my questions are:

1- What is the main reason/reasons for having warped/curved 3D point clouds? is it camera calibration only or other steps may introduce error as well? how can I check on that?

2- Can you suggest another camera calibration toolbox besides MATLAB's and Caltech's? maybe one that is more suitable for radial distortions?

Thanks

simillar question

coordinate system

Answer 1

[Dima Lisin]

Hi Luca,

Thanks for your kind words.

You are correct about the negative values. If camera 1 is on the right, then that is what you would get.

As for the curving of the point cloud, that is harder to diagnose. I have some questions for you. How many image pairs did you use for calibration? What kind of reprojection errors did you get? What is the resolution of your camera? When you undistorted the images, did they look reasonable to your eyes? Or was there still some distortion present?

What I am getting at is that you may need to re-calibrate using more images with the checkerboard close to the edges of the image, to get a better estimate of the distortion coefficients.

Another question: what 'OutputView' did you use for the undistortImage? And if you used 'full' or 'valid', did you remember to adjust your points for the shift of the origin?

By the way, next time you have a question on this topic, please specify "Computer Vision System Toolbox" under "Products". This way you will probably get an answer more quickly.

[Dima Lisin]

Hi Luca,

Your code looks right. The problem seems to be in the calibration. The fact that you get a warped image with 3 coefficients tells me that you may not have enough data points close to the edges of the image to estimate the distortion accurately. It is hard to see with your images, though. If you take a picture of a scene with many straight edges and undistort it, you would get a better idea.

So I would still recommend taking more images with the checkerboard as close to the edges of the image as you can get. See if that helps.

Another thing to look at would be estimation errors. In R2014b the Stereo Camera Calibrator app can optionally return the standard error values for each of the estimated parameters. Those can give you confidence intervals and tell you whether you may need more data points. See this example.

Oh, and also make sure that your calibration images are not saved as jpeg. Please use a lossless format like tiff or png.

[Luca]

I just tried recalibrating the cameras with new images. This time I took the images with the checkerboard close to the edges of the image. I still got warped pointclouds! However this time with the new calibration images the undsitorted images are not warped. So I am really confused now. How can I know which one is better? What is the source of warped point clouds besides camera calibration? Here is a comparison between old and new calibration. As you can see the results are not within confidence interval of each other. Also I have changed the checkerboard pattern in the new calibration. Also I took 48 pairs of images but out of them only 16 pairs were finally used for calibration. The app rejected 16 and then I removed 16 later because the x and y direction was wrong and it was returning error in the range of 50 pixels! The final mean error was 0.15 pixels.

Old calibration with 3 radial distortion coefficients and tangential and skew:

Camera 1 Intrinsics

Focal length (pixels): [ 1052.0002 +/- 1.1178 1050.8927 +/- 1.1098 ] Principal point (pixels):[ 651.6718 +/- 1.9753 479.9664 +/- 1.4113 ] Skew: [ -0.8393 +/- 0.2492 ] Radial distortion: [ -0.3646 +/- 0.0072 0.2306 +/- 0.0433 -0.1585 +/- 0.1039 ] Tangential distortion: [ 0.0010 +/- 0.0002 -0.0004 +/- 0.0006 ]

Camera 2 Intrinsics

Focal length (pixels): [ 1052.7896 +/- 1.1249 1051.3245 +/- 1.1374 ] Principal point (pixels):[ 627.1975 +/- 1.8436 465.8775 +/- 1.5630 ] Skew: [ 0.1097 +/- 0.1487 ] Radial distortion: [ -0.3360 +/- 0.0139 -0.4400 +/- 0.2850 4.2018 +/- 1.6752 ] Tangential distortion: [ -0.0000 +/- 0.0002 0.0001 +/- 0.0002 ]

Position And Orientation of Camera 2 Relative to Camera 1

Rotation of camera 2: [ 0.9999 +/- 0.0015 0.0073 +/- 0.0020 0.0101 +/- 0.0002 ] Translation of camera 2 (mm): [ -120.1781 +/- 0.0323 0.3933 +/- 0.0277 0.7251 +/- 0.2750 ]

New camera calibration with 3 radial distortion coefficients and tangential and skew:

Camera 1 Intrinsics

Focal length (pixels): [ 1076.3247 +/- 10.4440 1076.5068 +/- 10.3557 ] Principal point (pixels):[ 667.2485 +/- 6.4486 493.7571 +/- 5.0526 ] Skew: [ 0.2786 +/- 0.5784 ] Radial distortion: [ -0.3994 +/- 0.0091 0.2895 +/- 0.0389 -0.1579 +/- 0.0565 ] Tangential distortion: [ 0.0000 +/- 0.0006 -0.0010 +/- 0.0009 ]

Camera 2 Intrinsics

Focal length (pixels): [ 1078.9902 +/- 10.5757 1078.3900 +/- 10.4780 ] Principal point (pixels):[ 650.3242 +/- 6.5206 477.2131 +/- 5.2568 ] Skew: [ 0.6446 +/- 0.5203 ] Radial distortion: [ -0.4029 +/- 0.0090 0.3016 +/- 0.0331 -0.1737 +/- 0.0401 ] Tangential distortion: [ -0.0007 +/- 0.0006 -0.0015 +/- 0.0009 ]

Position And Orientation of Camera 2 Relative to Camera 1

Rotation of camera 2: [ 1.0000 +/- 0.0007 0.0070 +/- 0.0013 0.0029 +/- 0.0002 ] Translation of camera 2 (mm): [ -117.2770 +/- 0.1127 0.3455 +/- 0.0668 -0.6153 +/- 0.5555 ]

[Dima Lisin]

Hi Luca,

In your new calibration, the estimation errors for the focal length and the principal point are very high... That usually means not enough points.

Why were x and y directions wrong in the images you excluded? Are you using an asymmetric checkerboard where one dimension is even and one odd? You definitely want to end up with more than 16 pairs of images in this case.

[Luca]

You are right. The error is high. No I was using a 5 by 5 checkerboard. The link to the checkerboard pattern in your stereo calibrator app example is broken. Can you please check on that? I do not have a checkerboard pattern from Mathwork. I will try again with Matlab camera calibrator app's checkerboard pattern. I guess I can use Caltech's pattern and crop it to an even by odd.

[Dima Lisin]

Hi Luca,

Are you trying to open the checkerboard pdf from the web, or from the documentation that came with your MATLAB installation? I am asking, because it will not work on the web. The pdf comes with your installation, and can only be accessed from within MATLAB's doc browser.

[Luca]

Got it, Thanks!

[Luca]

Hi Dima,

I just tried recalibrating the cameras with the new pattern in MATALB's doc browser. This time I took 40 images from different distances and with the pattern in different corners of the image as well as rotated. I tried 2 radial coefficients, 3 coeff, and tangential and skew. mean reprojection error was 0.14 pixels. The error in focal length and principal points were also low (less than 0.7). The undistorted images looked OK. However the point clouds are still warped! here is the calibration results:

Camera 1 Intrinsics ------------------- Focal length (pixels): [ 1049.1326 +/- 0.8869 1049.4288 +/- 0.8925 ] Principal point (pixels):[ 652.4303 +/- 0.7375 476.8087 +/- 0.4685 ] Skew: [ 0.1303 +/- 0.0717 ] Radial distortion: [ -0.3692 +/- 0.0008 0.2155 +/- 0.0027 -0.0894 +/- 0.0030 ] Tangential distortion: [ 0.0007 +/- 0.0001 0.0001 +/- 0.0001 ]

Camera 2 Intrinsics ------------------- Focal length (pixels): [ 1050.5841 +/- 0.8770 1050.5170 +/- 0.8846 ] Principal point (pixels):[ 634.2544 +/- 0.7344 461.6343 +/- 0.4741 ] Skew: [ 0.1113 +/- 0.0688 ] Radial distortion: [ -0.3682 +/- 0.0010 0.2156 +/- 0.0040 -0.0902 +/- 0.0055 ] Tangential distortion: [ 0.0001 +/- 0.0001 -0.0004 +/- 0.0001 ]

Position And Orientation of Camera 2 Relative to Camera 1 --------------------------------------------------------- Rotation of camera 2: [ 1.0000 +/- 0.0001 0.0076 +/- 0.0002 0.0040 +/- 0.0000 ] Translation of camera 2 (mm): [ -121.0488 +/- 0.0129 0.3705 +/- 0.0085 0.1717 +/- 0.0652 ]

here is a link to the calibration images I took. I am not sure what else to do about this problem ...

calibration images

[Dima Lisin]

Hmm... Your calibration looks excellent... Sounds like a good time to look at the subsequent steps.

How are you doing your reconstruction? How are you matching the features for triangulate? If you are doing dense reconstruction, how are you doing the rectification? How are you computing disparity?

[Luca]

Hi Dima,

My code is attached in the first comment above. I am doing sparse 3D reconstruction. So Image undistortion using the camera calibration files, feature detection using SURF, feature extraction, feature matching, and triangulation. All of the above steps are done using MATLAB's CV toolbox. So I don't think there are mistakes in them. Last time we checked you confirmed the code is OK (in the comments above). We agreed to recalibrate the cameras with the checkerboard in the the corners of the image. I did that and the point clouds are still warped...

[Luca]

Hi Dima,

I did some research and apparently Bumblebee cameras are known better to be calibrated with 5 or 6 radial distortion coefficients. MATLAB CV toolbox only has 3 radial coefficients. Do you know if I can change the code to get 6 coefficients? If not, do you know of any other camera calibration toolbox that has that option? I know Caltech and OpenCV has 3 coefficients as well. I think that is the main reason causing warped point clouds since my Dense reconstruction point clouds are warped too and the only thing these two methods have in common are the images and calibration.

[Dima Lisin]

This sounds like a different distortion model. Typically you have 3 radial distortion coefficients and, optionally, 2 tangential distortion coefficients.