try first using the detectCheckerboardPoints from matlab for each set of images (check the matlab manual) and then when you have the points use : imagePoints = cat(4, poitnsRGB, pointsIR); Then feed these to the estimateCameraParameters (check the manual for more info).
You may be able to calibrate your cameras using the estimateCameraParameters function. It takes detected image points from each camera irrespective of the sizes of the images.
The rectifyStereoImages function only supports images of the same size. Use undistortImage to undistort the two images.