This is the way, the reprojection error is computed:
-If you have a pair of images (No Calibration), then you have the camera matrix already. In such case, you use the triangulation using both cameras to find the point X(x,y,z) in space corresponding to the points x and x' in the left and right images respectively. Then, you use each camera P and P' to reproject the world point X back to the image to be x_hat and x'_hat respectively. Usually if the matches are correct ( and the cameras are correct as well), then the reprojection error ( which is the difference between x and x_hat) should be small, if it is large, it means that either the camera matrix is incorrect or the matches are incorrect.
- If you are calibrating the camera, then you have the coordinates of the image points (x) and the coordinates of the world points X, and you need to find the optimal camera P. Then you use the camera P as an operator on X to find the value x_hat=PX ( you project the world point into the image plane using the camera matrix). Again, the difference between (x and x_hat ) is the reprojection error. If it is small, it means the camera matrix ( that is estimated) is correct. If it is large it means that the camera matrix is wrong.