Best Practices for 3-D Reconstruction Using Structure from Motion

R2026b

Structure from motion (SfM) estimates the 3-D structure of a scene and camera poses from a set of calibrated 2-D images. This topic describes best practices for achieving reliable 3-D reconstructions using the sfm object. The SfM pipeline consists of these stages:

1. Prepare input data — Capture images with sufficient overlap and texture, then calibrate the camera to obtain accurate intrinsics.

2. Build the view graph — Connect visually similar image pairs using connectImagePairs.

3. Verify image pair geometry — Refine the view graph using geometric constraints with verifyImagePairs.

4. Initialize reconstruction — Select a robust initial view pair and triangulate the first 3-D points using triangulateInitialViews.

5. Reconstruct incrementally — Incrementally process all remaining views and reconstruct the full 3-D scene using reconstruct.

The quality of results depends on both the input data and how each stage of the pipeline is configured. If your reconstruction fails or produces poor results, see Troubleshooting Common Issues for a quick-reference guide.

Prepare Input Data for SfM Pipeline

The sfm object requires undistorted images and accurate camera intrinsics as input. Poor quality input data, such as blurry images, insufficient overlap, or inaccurate calibration, cannot be recovered through parameter tuning alone. Prepare your data carefully before running the pipeline.

• Accurate calibration — Calibrate camera before running SfM and undistort all images. Aim for reprojection error below 1 pixel. For more information, see Using the Single Camera Calibrator App.

• Sufficient overlap — Aim for 60-80% overlap between consecutive images so that each point is observed from multiple viewpoints.

  Ensure Sufficient Overlap

  Each scene point must appear in at least two images to be triangulated, but three or more views produce substantially more reliable results. Aim for 60-80% overlap between consecutive images so that each point in the scene is observed from multiple viewpoints.

  The required overlap depends on your capture strategy:

  • For sequential captures (walking or driving), take images frequently enough that consecutive frames share the majority of their content.

  • For orbital captures (around an object), move in an arc with small angular steps (10-15 degrees between shots).

  • For aerial or grid captures, use 70-80% frontal overlap and 50-60% side overlap between adjacent flight lines.

  Insufficient overlap causes the view graph to fragment into disconnected components and prevents the pipeline from reconstructing a complete model.

• Varied viewpoints — Move the camera between shots to create parallax. Rotation-only motion cannot be reliably triangulated.

  Capture Images from Varied Viewpoints

  SfM requires parallax to estimate depth. Parallax is the apparent shift of objects when the camera moves. Images taken from the same position with only rotation produce degenerate geometry and cannot be triangulated.

  • Move the camera between shots rather than only rotating it. Even a few centimeters of lateral translation helps.

  • For objects, capture from multiple heights and angles. Vary the elevation of the camera between passes.

  • Avoid taking many images from nearly identical positions. Redundant views add processing cost without improving accuracy.

  Capturing all images from the same distance and angle relative to a surface produces a planar point cloud with poor depth accuracy. Include oblique views (30-45 degrees from the surface normal) to strengthen triangulation geometry.

  Forward-only motion, as typically observed in driving scenarios on a straight road, is not a good candidate for SfM due to minimal parallax. For such workflows, consider using Visual SLAM instead. See How to Improve Accuracy in Visual SLAM.

• Sharp, well-lit images — Use a fast shutter speed, low ISO, and consistent illumination. Avoid moving objects in the scene.

  Avoid Motion Blur and Noisy Images

  Blurry images produce fewer and less accurate feature detections. This directly reduces the number of matches and increases the risk of tracking failure. To minimize image quality issues:

  • Use a fast shutter speed to freeze motion. Minimize the amount of motion blur across the image.

  • Keep ISO as low as possible to reduce noise.

  • If lighting is poor, use a wider aperture or add illumination rather than increasing ISO or reducing shutter speed.

  • If using video as input, prefer higher-resolution footage and extract frames at a reduced rate rather than using every frame.

  Control Lighting and Avoid Moving Objects

  Consistent lighting and a static scene are essential for reliable feature matching:

  • Capture all images under consistent illumination. Large brightness changes between views reduce feature matching reliability.

  • Avoid images shot directly toward a bright light source (e.g., against the sun). High dynamic range causes saturated or underexposed regions.

  • Minimize specular highlights on shiny or reflective surfaces. These highlights appear at different locations across views and create false correspondences.

  • Remove or avoid moving objects such as people or vehicles in the scene. Moving objects violate the rigid-scene assumption and introduce outlier correspondences that degrade the reconstruction.

• Good texture — Avoid featureless or repetitive surfaces. Add textured objects if needed.

  Choose Scenes with Good Texture

  Feature detection relies on local intensity variations. Surfaces with uniform color or repetitive patterns produce few or unreliable features. Keep these texture considerations in mind:

  • Avoid imaging scenes consisting primarily of blank walls, clear sky, smooth surfaces, or uniformly colored materials.

  • If your scene has large featureless regions, add background objects with distinguishing texture (posters, markers, or patterned objects) to provide anchor points.

  • Repetitive patterns (tiles, bricks, window grids) can produce false matches because multiple locations appear identical. Include enough surrounding context to disambiguate between repetitive patterns.

• Fixed focal length — Do not change zoom between captures. A single set of intrinsics must apply to all images.

  Use Fixed Focal Length

  Do not change the zoom between captures. A fixed focal length ensures a single set of intrinsic parameters applies to all images. If the focal length varies across images, the sfm object cannot apply a single set of intrinsics, and reconstruction accuracy degrades.

  Select Appropriate Image Resolution

  Higher resolution images produce more features and finer localization. However, they also increase memory usage and computation time. Beyond a certain point, additional resolution provides diminishing returns for reconstruction quality while significantly increasing processing cost.

  Because the sfm object stores all features, matches, and view graph edges in memory, resolution has a direct impact on memory consumption. If you encounter memory or processing time issues, consider downsampling your images while adjusting the intrinsics to match. If your images have distortion, undistort them before downsampling.

  scaleFactor = 0.5; % Downsample to half resolution
  for i = 1:numel(imds.Files)
      I = readimage(imds, i);
      I = imresize(I, scaleFactor);
      imwrite(I, imds.Files{i});
  end
  
  % Adjust intrinsics to match the downsampled resolution
  focalLength = intrinsics.FocalLength * scaleFactor;
  principalPoint = intrinsics.PrincipalPoint * scaleFactor;
  imageSize = round(intrinsics.ImageSize * scaleFactor);
  intrinsics = cameraIntrinsics(focalLength, principalPoint, imageSize);

  If you downsample too aggressively, feature detection produces too few keypoints for reliable matching. As a guideline, preserve enough resolution so that scene details you need to reconstruct remain visible at the pixel level.

Calibrate Camera and Remove Distortion

SfM requires accurate camera intrinsic parameters. Errors in focal length, principal point, or distortion coefficients propagate into every stage of the pipeline, causing incorrect pose estimates and distorted 3-D geometry.

Obtain Intrinsics Before Reconstruction

Calibrate your camera before running SfM. Use the Camera Calibrator app or estimateCameraParameters function for pinhole cameras. Use the estimateFisheyeParameters function for fisheye cameras. Follow these guidelines:

• Capture 10-20 calibration images following the guidelines in Data Collection Guidelines for Single-Camera Calibration.

• Cover the entire image frame with calibration pattern observations.

• Achieve a mean reprojection error below 1 pixel.

• Visually verify that distortion is properly corrected by ensuring that straight lines in the scene appear straight in the undistorted images, particularly near the image boundaries.

For more information, see Using the Single Camera Calibrator App.

Undistort Images Before Running Pipeline

The sfm object does not model lens distortion internally. Remove distortion from images before creating the sfm object:

intrinsics = cameraIntrinsics([fx fy], [cx cy], imageSize, ...
    RadialDistortion=[k1 k2], TangentialDistortion=[p1 p2]);

for i = 1:numel(imds.Files)
    I = readimage(imds, i);
    [I, undistortedIntrinsics] = undistortImage(I, intrinsics);
    imwrite(I, imds.Files{i});
end

After undistortion, use undistortedIntrinsics (a cameraIntrinsics object with zero distortion) with the sfm object.

Handle Fisheye and Wide-Angle Lenses

The sfm object assumes a pinhole camera model. If you are using a fisheye or wide-angle lens, you must remove the fisheye distortion before running SfM:

fisheyeParams = estimateFisheyeParameters(calibrationImages, ...);
for i = 1:numel(imds.Files)
    I = readimage(imds, i);
    [I, undistortedIntrinsics] = undistortFisheyeImage(I, fisheyeParams.Intrinsics);
    imwrite(I, imds.Files{i});
end

After undistortion, use the output intrinsics (a standard pinhole model) when creating the sfm object. Note that undistorting a fisheye image reduces the effective field of view. Ensure your images still have sufficient overlap after correction.

SfM Parameter Recommendations by Scene Type

Different capture scenarios require different trade-offs between data preparation and parameter tuning. Use this table to identify recommendations for your scenario.

Scene Type	Data Preparation	Parameter Tuning
Indoor	Frame shots to include textured objects (furniture, posters) Vary camera height between passes to create parallax along the vertical axis Indoor scenes often have limited texture on walls and ceilings, so add background objects if needed	Increase NumSimilarImages in connectImagePairs to capture more overlap across rooms Decrease MinNumInliers in verifyImagePairs if walls lack texture
Outdoor/Aerial	Remove motion-blurred frames Use consistent altitude Maintain 70-80% overlap in grid patterns Typical panorama photos with largely planar or distant scenes will not produce good reconstructions due to low parallax	Increase MaxReprojectionError in reconstruct to tolerate minor altitude variations Increase NumSimilarImages in connectImagePairs for sparse captures
Rotation-Dominant Motion (Turntable, Object Scanning)	Use 20-36 images per orbit (10-15° steps) Capture multiple orbits at different elevations Ensure textured surface because shiny, transparent, or uniform objects fail to produce stable features	Decrease MinMedianAngle in triangulateInitialViews since baselines are small Increase MaxTriangulationError in reconstruct if object is small relative to camera distance
Forward Motion Along Optical Axis (Corridors, Driving)	Skip frames to increase effective baseline Parallax is low, which makes initialization challenging Sequences benefit from loop closures where the camera revisits earlier viewpoints For driving scenarios, consider using visual SLAM with monovslam	Reduce MinMedianAngle in triangulateInitialViews to handle low parallax during initialization Decrease MinNumInliers in verifyImagePairs to retain more connections along the motion direction Decrease GlobalRefinementFrequency in reconstruct to reduce accumulated drift
Large Image Sets or Long Sequences	Avoid processing all images at once if many views are redundant or weakly connected Remove blurry, near-duplicate, or low-overlap images before running the pipeline If a single reconstruction does not register enough views, divide the sequence into overlapping spatial or temporal subsets	No single parameter change guarantees successful registration for very large datasets. Tune connectImagePairs, verifyImagePairs, and reconstruct within each subset independently. Combining subset reconstructions is an advanced workflow that requires custom alignment and validation. For more details, see Point Cloud Registration and SLAM (Point Cloud Toolbox).

Troubleshooting Common Issues

Improve the completeness of the 3-D reconstruction by ensuring all images are registered. Reconstruction accuracy improves significantly when no views are dropped. Start by tuning verifyImagePairs parameters to retain more geometrically valid connections in the view graph, which in turn allows reconstruct to register more views.

If your reconstruction still fails or produces unexpected results, use this table to identify the likely cause and remedy.

Symptom	Likely Cause	Remedy
View graph is disconnected	Images lack visual overlap or features	Adjust connectImagePairs parameters: Increase NumSimilarImages Decrease MinNumMatches Recapture with more overlap
Most edges removed by verifyImagePairs	Planar scene or low parallax	Adjust verifyImagePairs parameters: Decrease MinNumInliers Increase MaxDistance Add oblique views
triangulateInitialViews fails	No pair has sufficient parallax	Adjust triangulateInitialViews parameters: Decrease MinMedianAngle Increase MaxTriangulationError Verify camera translation between views
Few images registered	Weak view graph connections	Adjust reconstruct parameters: Increase MaxReprojectionError Decrease MinNumInliers
Visible drift	Accumulated errors without global correction	Adjust reconstruct parameters: Decrease GlobalRefinementFrequency Increase MaxSolverIterations
Flat or degenerate point cloud	Planar scene or no camera translation	Add oblique image captures Include 3-D depth variation
High reprojection error	Inaccurate calibration or blurry images	Recalibrate and undistort images Reduce motion blur
Few features detected or poor matching on overexposed/underexposed images	Poor image contrast reduces feature detection quality	Preprocess images to improve contrast before running the pipeline: Apply adapthisteq (CLAHE) for localized contrast enhancement Use histeq for global histogram equalization Use imadjust to remap intensity range Alternatively, relax MinNumInliers in verifyImagePairs and increase MaxTriangulationError in reconstruct
Noisy or scattered points	Weak triangulation angles	Adjust reconstruct parameters: Decrease MaxTriangulationError Increase MinTriangulationAngle

If parameter tuning does not resolve the issue, consider mapAnything object as an alternative that uses a pretrained feed-forward transformer model which does not rely on explicit feature matching and can handle challenging scenes where sfm struggles to register all images. For more guidance on choosing between the two approaches, see Choose Between SfM and MapAnything.

Tune Individual SfM Pipeline Stages

Use this section for finer control over individual stages of the SfM pipeline. Each subsection covers a specific stage with detailed parameter tuning guidance in expandable sections. For quick guidance, refer to the troubleshooting table and scene type recommendations above.

Build View Graph

The connectImagePairs function builds a view graph by finding visually similar images using SIFT features and bag-of-words retrieval. A well-connected graph is the foundation for all subsequent stages. Expand the sections below for more information on parameter tuning, connectivity verification, and custom vocabulary options.

Adjust Connectivity Parameters

If the view graph is disconnected or too sparse, adjust these name-value arguments in the connectImagePairs function:

• Increase NumSimilarImages when your dataset has many images and moderate overlap. The default value of 10 may miss valid connections in large collections. In practice, NumSimilarImages typically has the most significant effect on view graph connectivity.

• Decrease MinNumMatches when images have few features (low texture or small images). The default value of 15 requires at least 15 raw feature matches per pair.

• Decrease MaxRatio (ratio test threshold) to be more conservative about feature matches when scenes contain repetitive textures. For more information about feature matching, see matchFeatures.

% Increase search breadth for datasets with many images or weak overlap
sfmObj = connectImagePairs(sfmObj, NumSimilarImages=20, MinNumMatches=10);

Verify Graph Connectivity

After calling connectImagePairs, visualize the similarity matrix to identify gaps in coverage for sequentially captured imagery. Dense blocks along the diagonal represent sequential image overlap. Off-diagonal entries correspond to loop closures or non-sequential connections. Rows or columns with all zeros identify disconnected images that the pipeline cannot reconstruct.

imagesc(sfmObj.SimilarityMatrix)
colorbar; axis image
title("Similarity Matrix for Connected Image Pairs")

For unordered imagery, visualizing the view graph as nodes and edges of a directed graph (digraph) is more meaningful.

G1 = createPoseGraph(sfmObj);
plot(G1, 1:sfmObj.NumImages, Layout="circle")
title("View Graph for Connected Image Pairs")

Ensure a connected graph where every node is reachable from every other node by following edges. In other words, there are no disconnected components in the view graph. For example, the plot on the left shows three disconnected components resulting from a lower NumSimilarImages value, while the plot on the right shows a well-connected graph resulting from a higher NumSimilarImages value.

After calling connectImagePairs, you can check whether the view graph is connected programmatically:

% Check if graph is connected
G = createPoseGraph(sfmObj);
isGraphConnected = max(conncomp(G, 'Type', 'weak')) == 1; % Returns true (1) if single connected component, false (0) if multiple

% Retry with relaxed parameters
if ~isGraphConnected
    sfmObj = connectImagePairs(sfmObj, NumSimilarImages=25, MinNumMatches=8);
end

Use Custom Bag-of-Features Vocabulary

The connectImagePairs function uses a default SIFT-based DBoW2 vocabulary to identify visually similar image pairs. The default vocabulary is trained on a general-purpose dataset and works well for common scenes. However, if your images come from a specialized domain such as aerial imagery, medical imaging, industrial inspection, or underwater scenes, the default vocabulary may not represent the visual features in your dataset effectively. This may lead to missed connections in the view graph despite tuning parameters for the connectImagePairs function.

In such cases, train a custom vocabulary on your own images or on a representative dataset from your domain using the bagOfFeaturesDBoW object. The custom vocabulary must use SIFT features because connectImagePairs extracts SIFT features internally.

% Train a custom SIFT vocabulary from your images
bag = bagOfFeaturesDBoW(imds, FeatureType="SIFT");

% Use the custom vocabulary when building the view graph
sfmObj = connectImagePairs(sfmObj, CustomBagOfFeatures=bag);

Verify Image Pair Geometry

The verifyImagePairs function removes false connections by enforcing epipolar constraints. The goal is to retain geometrically consistent pairs while removing spurious matches. Expand the sections below for more information on threshold tuning and handling planar scenes.

Tune Geometric Verification Parameters

Adjust verification thresholds based on whether you are losing too many valid connections or retaining too many false ones.

• If too many connections are removed and the graph becomes disconnected, decrease MinNumInliers or increase MaxDistance:

sfmObj = verifyImagePairs(sfmObj, MinNumInliers=20, MaxDistance=6);

• If false pairs survive and introduce errors in later stages, increase MinNumInliers or decrease MaxDistance:

sfmObj = verifyImagePairs(sfmObj, MinNumInliers=50, MaxDistance=2);

Handle Planar Scenes

Scenes with dominant planes (walls, floors, tabletops) can cause the essential matrix estimation to degenerate. verifyImagePairs automatically selects between an essential matrix for general 3-D scenes and a homography for planar geometry. However, if planar geometry dominates, initialization may produce a flat point cloud. To mitigate this:

• Include oblique views that observe non-planar scene structure.

• Ensure at least some 3-D depth variation is visible in the initial view pair.

Initialize Reconstruction

The triangulateInitialViews function selects the best image pair and triangulates the seed 3-D points on which the entire incremental reconstruction builds. A poor initialization propagates errors into all subsequent views. Expand the sections below for more information on threshold adjustment, diagnostics, and manual pair selection.

Adjust Initialization Thresholds

For most datasets, the defaults (MinMedianAngle=16, MaxTriangulationError=4) work well. Adjust these only when initialization fails.

% Relax for forward-motion sequences with small baselines
[sfmObj, info] = triangulateInitialViews(sfmObj, MinMedianAngle=5, MaxTriangulationError=6);

% Tighten for high-quality datasets with good parallax between views
[sfmObj, info] = triangulateInitialViews(sfmObj, MinMedianAngle=20, MaxTriangulationError=2);

Interpret Initialization Diagnostics

Use the info output to assess initialization quality and identify potential issues with the initial pair.

• A median triangulation angle below 5 degrees indicates a weak baseline. The initial points may be unreliable and can cause drift in later stages.

• A high mean reprojection error (relative to the MaxTriangulationError threshold) indicates imprecise calibration or poor feature localization.

[sfmObj, info] = triangulateInitialViews(sfmObj, Verbose=true);
disp(info.MedianTriangulationAngle)
disp(info.MeanReprojectionError)

Specify Initial Pair Manually

If automatic selection fails or produces poor results, specify a view pair that you know has good overlap and sufficient baseline. Choose a pair where:

• The two images share substantial visual overlap (many matched features).

• There is noticeable camera translation between the two views.

• The scene contains 3-D depth variation (not a single flat surface).

sfmObj = triangulateInitialViews(sfmObj, [viewID1 viewID2]);

Reconstruct Incrementally

The reconstruct function incrementally registers all remaining views. The key challenge is ensuring all images register while maintaining geometric accuracy. Expand the sections below for more information on registering more images, improving triangulation quality, speeding up processing, and monitoring reconstruction progress.

Register More Images

The most common reason for missing images is that no view has enough 2-D to 3-D correspondences. Relax the registration criteria to allow views with fewer correspondences.

sfmObj = reconstruct(sfmObj, MaxReprojectionError=15, MinNumInliers=15);

Improve Triangulation Quality

Tighten the triangulation thresholds to admit only well-conditioned 3-D points.

sfmObj = reconstruct(sfmObj, MaxTriangulationError=2, MinTriangulationAngle=3);

Increase solver iterations for more precise bundle adjustment.

sfmObj = reconstruct(sfmObj, MaxSolverIterations=50);

Speed Up Processing

Reduce the scope of local bundle adjustment.

sfmObj = reconstruct(sfmObj, NumViewsRefined=4, MaxNumRefinement=1, GlobalRefinementFrequency=20);

This trades accuracy for speed. Use larger values of GlobalRefinementFrequency to perform global bundle adjustment less often.

Monitor Reconstruction Progress

Use Verbose=true to track which views are registered and identify failure points.

sfmObj = reconstruct(sfmObj, Verbose=true);

Check the processing order in sfmObj.ProcessedImages. Views with strong overlap should be processed early. If well-overlapping views appear late or not at all, the feature matching or pose estimation thresholds may be too strict.