Main Content

Get Started with Lidar Camera Calibrator

The Lidar Camera Calibrator app enables you to interactively perform calibration between a lidar sensor and a camera by estimating a rigid transformation between them.

Lidar Camera Calibrator App Session

This topic shows you the Lidar Camera Calibrator app workflow, as well as features you can use to analyze and improve your results. The first and the most important part of the calibration process is to obtain accurate and useful data. For guidelines and tips for capturing data, see Calibration Guidelines.

Load Data

To open the Lidar Camera Calibrator app, enter this command in the MATLAB® command prompt.

lidarCameraCalibrator

Alternatively, you can open the app from the Apps tab, under Image Processing and Computer Vision.

The app opens to an empty session. The app reads point cloud data in the PLY and point cloud data (PCD) formats, and images in any format supported by imformats. If your data is stored in a rosbag file, see the Read Lidar and Camera Data from Rosbag File tutorial to convert it accordingly.

Load the calibration data into the app.

  1. On the app toolstrip, select Import > Import Data, this opens the Import Data dialog box.

  2. In the Folder for images box, enter the path to the folder that contains the image files you want to load. Alternatively, select Browse to navigate to the folder containing the images,and click Select Folder.

  3. In the Folder for point clouds box, enter the path to the folder that contains the sequence of PCD or PLY files you want to load. Alternatively, select the Browse to navigate to the folder containing the files,and click Select Folder.

  4. In the Checkerboard Settings section, enter the calibration checkerboard parameters. Specify the size for each checkerboard square in the Square Size box, and select the units of measurement from the list next to the box.

  5. In the Padding box, enter the padding values for the checkerboard. For more information on padding, see Checkerboard Padding. Click OK to import your data.

The input image and point cloud files must have the same names. The app uses file names to pair the image and point cloud data. It compares images to the corresponding point clouds with the same file name.

Tip

To add more images and point clouds to the session at any point in the session, select Import > Add Data to Session.

Detect Features

After you load the image and point cloud data, the app performs an automatic feature detection pass on them. The app detects checkerboard corners from the image data and the checkerboard plane from the point cloud data using the specified checkerboard parameters.

Lidar Camera Calibrator Calibration Tab

When the app detects checkerboard features in both the image and point cloud, it displays them in the Accepted Data pane, and accepts the image and point cloud data pair for calibration.

When the app does not detect features in the image, the point cloud, or both, it displays them in the Rejected Data pane. You can use the Select Region of Interest or Select Checkerboard Region tool to detect features in the rejected data.

Select a data pair from the Accepted Data or Rejected Data pane in the data browser to display it in the visualization pane.

You can use these tools in the Action section of the app toolstrip to work with the accepted and rejected data pairs.

  • Move to Accepted Data — Move a data pair from rejected data to accepted data.

  • Move to Rejected Data — Move a data pair from accepted data to rejected data.

  • Remove — Delete the selected data pair.

For a list of keyboard shortcuts to use in the data browser, see Data Browser.

Load Camera Intrinsic Parameters

By default, the app internally computes camera intrinsic parameters from the input image data to perform feature detection. You can also load camera intrinsic parameters into the app by, in the Camera Intrinsics section of the app toolstrip, selecting Use Fixed Intrinsics. In the dialog box that opens, specify the location of your camera intrinsic parameters and load them into the app. You can load intrinsic parameters from a file or from the MATLAB workspace. Then, in the Detect Features section, select Detect to re-detect features in the input data with the new intrinsic parameters.

To reset camera intrinsic parameters to the default values computed by the app, select Compute Intrinsics. To load different intrinsics values, select Use Fixed Intrinsics and then Load Intrinsics.

Select Region of Interest

Specifying a region of interest in which to detect features can improve your feature detection results, especially in Rejected Data. To specify a region of interest, first, from the app toolstrip, select Edit ROI. This opens the Edit ROI tab.

Edit ROI Tab

In the Edit ROI tab, the app displays point cloud data with an ROI cuboid. Adjust the ROI cuboid to more closely match the region containing the checkerboard. This reduces data rejections and improves performance by containing feature detection to a specific region.

Use these steps to fine-tune the checkerboard detections in your point cloud data using Edit ROI.

  1. Select any data pair. You can select a rejected data pair to tune the ROI.

  2. To adjust the ROI, in the point cloud pane on the Edit ROI tab, pause on the ROI cuboid. The pointer becomes a hand symbol. Click any side of the ROI and drag to adjust the cuboid size. To lock the cuboid dimensions, right click on the cuboid and select Lock Dimensions. You can also undo and redo your modifications to the ROI.

  3. Select Snap To ROI to visualize the checkerboard points within the ROI cuboid, and adjust the cuboid size accordingly. To view the whole point cloud, clear Snap To ROI.

    You can also use XY View, YZ View, and XZ View options to visualize top view, front view, and side view of a point cloud scene, respectively.

  4. You can update the ROI for each individual frame or for all frames at once.

  5. Select Apply to save your changes, or Cancel to discard them.

After updating the ROI for the point cloud frames, on the Calibration tab of the app toolstrip, in the Detect Features section, select Detect to re-detect features in the input data within the selected ROI.

Tip

Use keyboard shortcuts to perform these tasks more interactively. For Edit ROI keyboard shortcuts, see Edit ROI.

Select Checkerboard Region

To further tune feature detections, click Select Checkerboard on the app toolstrip. The app opens the Select Checkerboard tab, where you can manually select checkerboard points in any point cloud frame.

Select Checkerboard Tab

In the Select Checkerboard tab,

  1. Select any data pair. You can select a rejected data pair and find the checkerboard in the point cloud.

  2. Use the zoom and rotate options in the axes toolbar of the point cloud display to locate the checkerboard. You can also use XY View, YZ View, and XZ View options to visualize top view, front view, and side view of a point cloud scene, respectively.

  3. Click Select Checkerboard, the cursor changes into a crosshair.

  4. Click and drag the cursor over the checkerboard. A selection rectangle appears, with the points inside it highlighted in red. Alternatively, you can also select Brush/Select Data on the axes toolbar.

  5. After selecting the points, rotate the point cloud to check whether any background points have been selected. If your selection contains unwanted points, select Erase to start over.

  6. Select Apply to save the selected points, or Cancel to discard them.

This checkerboard selection applies only to the selected point cloud. After updating the checkerboard points, select Detect to re-detect features in the input data with the updated checkerboard points.

Feature Detection Settings

The app provides these feature detection settings using which you can tune the detection parameters.

Feature Detection Settings

  • Remove Ground — Remove ground points from the point cloud. The app uses the pcfitplane function to estimate the ground plane. The Remove Ground feature is enabled by default. Select Remove Ground to clear it.

  • Cluster Threshold — Clustering threshold for two adjacent points in the point cloud, specified in meters. The clustering process is based on the Euclidean distance between adjacent points. If the distance between two adjacent points is less than the clustering threshold, both points belong to the same cluster. Low-resolution lidar sensors require a higher Cluster Threshold, while high-resolution lidar sensors benefit from a lower Cluster Threshold.

  • Dimension Tolerance — Tolerance for uncertainty in the rectangular plane dimensions, specified in the range [0,1]. A higher Dimension Tolerance indicates a more tolerant range for the rectangular plane dimensions.

After updating new detection parameters, select Detect to re-detect features in the input data.

Calibration

When you are satisfied with the detection results, select Calibrate button to calibrate the sensors. If you have an estimated transformation matrix, select Initial Transform to load the transformation matrix from a file or the workspace. The app assumes the rotation angle between the lidar sensor and the camera is in the range [–45 45 ], in degrees, along each axis. For a rotation angle outside this range, use Initial Transform to specify an initial transformation to improve calibration accuracy.

After calibration, the app interface displays the image with the checkerboard points from the point cloud projected onto it. The app uses the projectLidarPointsOnImage function to project the lidar points onto the image. The color information of the images is fused with the point cloud data using the fuseCameraToLidar function.

Checkerboard point cloud points projected onto the image, and color information fused with the point cloud

The app also provides the inaccuracy metrics for the transformation matrix using error plots. The plots specify these errors in each data pair:

  • Translation Errors — The difference between the centroid coordinates of the checkerboard planes in the point clouds and those in the corresponding images. The app returns the error values in meters.

  • Rotation Errors — The difference between the normal angles defined by the checkerboard planes in the point clouds and those in the corresponding images. The app estimates the plane in the image using the checkerboard corner coordinates. The app returns the error values in degrees.

  • Reprojection Error— The difference between the projected (transformed) centroid coordinates of the checkerboard planes from the point clouds and those in the corresponding images. The app returns the error values in pixels.

Error Plots

When you select a data pair in the data browser, the corresponding bars in the error plot are highlighted in dark blue. You can tune the calibration results by removing outliers. Drag the red line on each plot vertically to set error limits. The app selects all the data pairs with an error value greater than error limit as outliers, and highlights the error bars and their corresponding data pairs in the data browser in blue. Right-click any of the selected data pairs on the data browser and select Remove and Recalibrate to delete the outliers and recalibrate the sensors. Deleting outliers can improve calibration accuracy. For a list of keyboard shortcuts to use with the error plots, see Error Plots.

Export Results

Export options for Lidar Camera Calibrator

You can export the transformation matrix and error metrics, as variables, into the workspace or a MAT-file. You can generate a MATLAB script of the complete app workflow to use in your projects.

Keyboard Shortcuts and Mouse Actions

Note

On Macintosh platforms, use the Command (⌘) key instead of Ctrl.

Use keyboard shortcuts and mouse actions to increase productivity while using the Lidar Camera Calibrator app.

Data Browser

TaskAction
Navigate through data pairs in the Accepted Data or Rejected Data paneUp or down arrow
Select all data pairs in the data browserCtrl+A
Select multiple data pairs above or below the currently selected data pair.Hold Shift and press the up arrow or down arrow

Select multiple data pairs.

Hold Ctrl and click on data pairs
Delete the selected data pair from the data browser.
  • PC: Backspace or Delete

  • Mac: delete

A dialog box appears to deletion.

Select the data pairN above the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.
  • PC: Page Up

  • Mac: Hold Fn and press the up arrow

Select the data pairN below the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.
  • PC: Page Down

  • Mac: Hold Fn and press the down arrow

Select the first data pair in the data browser.
  • PC: Home

  • Mac: Hold Fn and press the left arrow

Select the last data pair in the data browser.
  • PC: End

  • Mac: Hold Fn and press the right arrow

Error Plots

Use these shortcuts on the error plots to analyze the data. Operations on any of the three error plots affect the corresponding error bars on all three plots.

TaskAction
Select the error bar left of the currently selected error bar.Left arrow
Select the error bar right of the currently selected error bar.Right arrow
Select the error bar right or left of the currently selected error bar, in addition to the currently selected error bar.Hold Shift and press the left arrow or right arrow

Select multiple error bars.

Hold Ctrl and click error bars
Select all error bars.Ctrl+A
Delete the selected error bar and corresponding data pair from the data browser. The app then recalibrates the sensors.
  • PC: Backspace or Delete

  • Mac: delete

A dialog box appears to confirm deletion.

Edit ROI

Shortcuts to use on the Edit ROI tab.

TaskAction

Undo ROI size change.

Note

The app stores only the last three sizes of the ROI, so you cannot undo more than three times in a row.

Ctrl+Z

Redo ROI size change.

Ctrl+Y
Clear ROI selectionEsc

Limitations

The Lidar Camera Calibrator app has these limitations:

  • The script generated from Export > Generate MATLAB Script does not include any checkerboard regions manually selected using the Select Checkerboard feature. In the script, the checkerboard region is detected in the specified ROI.

  • After manually selecting checkerboard regions using the Select Checkerboard feature, when you return to the Calibration tab, you can see the selected points (highlighted in red) only while viewing the whole point cloud (when Snap To ROI is cleared).

See Also

| | | | | |

Related Examples

More About