What is the scientific reason for removing high reprojection error images from the stereo calibration app?

In the stere camera calibration app page (https://www.mathworks.com/help/vision/ug/stereo-camera-calibrator-app.html) there is a statemente that says:

Consider removing image pairs if the images:

• Have a high mean reprojection error.
• Are blurry.
• Contain a checkerboard at an angle greater than 45 degrees relative to the camera plane.
• Contain incorrectly detected checkerboard points.

What is the scientific reason to do so? I mean if the image is sharp, corners are identified correctly, the checkerboard is not at an angle greater than 45 degrees and we have an high reprojection error, why we should remove this image/couple of images anyway?

Answer 1

[hani mohammed]

This is the way, the reprojection error is computed:

-If you have a pair of images (No Calibration), then you have the camera matrix already. In such case, you use the triangulation using both cameras to find the point X(x,y,z) in space corresponding to the points x and x' in the left and right images respectively. Then, you use each camera P and P' to reproject the world point X back to the image to be x_hat and x'_hat respectively. Usually if the matches are correct ( and the cameras are correct as well), then the reprojection error ( which is the difference between x and x_hat) should be small, if it is large, it means that either the camera matrix is incorrect or the matches are incorrect.

- If you are calibrating the camera, then you have the coordinates of the image points (x) and the coordinates of the world points X, and you need to find the optimal camera P. Then you use the camera P as an operator on X to find the value x_hat=PX ( you project the world point into the image plane using the camera matrix). Again, the difference between (x and x_hat ) is the reprojection error. If it is small, it means the camera matrix ( that is estimated) is correct. If it is large it means that the camera matrix is wrong.

Hani

Answer 2

[Daniel Adam]

Hi,

I would say there is no scientific reason. It is just more common that the error of stereo calibration is caused by blurred chessboard images or misdetections. If most of your reprojection errors are 0.5pxl and two of them are 45pxl, then you can be pretty sure it is because of bad quality of those two images/detections. As always, it is about using reasonable threshold, check the distribution of errors, do some expertize-based decision. That's why documentation says "Consider removing image pairs if...".