This example shows how to create a mosaic from a video sequence. Video mosaicking is the process of stitching video frames together to form a comprehensive view of the scene. The resulting mosaic image is a compact representation of the video data, which is often used in video compression and surveillance applications.
|On this page…|
This example illustrates how to use detectFASTFeatures, extractFeatures, matchFeatures, and estimateGeometricTransform to create a mosaic image from a video sequence. First, the example identifies the corners in the first (reference) and second video frames. Then, it calculates the affine transformation matrix that best describes the transformation between corner positions in these frames. Finally, the example overlays the second image onto the first image. The example repeats this process to create a mosaic image of the video scene.
Define the size and location of the output mosaic image.
[w, h] = deal(680, 400); % Size of the mosaic [x0, y0] = deal(-5, -60); % Upper-left corner of the mosaic xLim = [0.5, w+0.5] + x0; yLim = [0.5, h+0.5] + y0; outputView = imref2d([h,w], xLim, yLim);
Create a VideoFileReader System object to read video from a file.
hsrc = vision.VideoFileReader('vipmosaicking.avi', 'ImageColorSpace', ... 'RGB', 'PlayCount', 1);
Create a AlphaBlender System object to overlay the consecutive video frames to produce the mosaic.
halphablender = vision.AlphaBlender( ... 'Operation', 'Binary mask', 'MaskSource', 'Input port');
Create two VideoPlayer System objects, one to display the corners of each frame and other to draw the mosaic.
hVideo1 = vision.VideoPlayer('Name', 'Corners'); hVideo1.Position(1) = hVideo1.Position(1) - 350; hVideo2 = vision.VideoPlayer('Name', 'Mosaic'); hVideo2.Position(1) = hVideo1.Position(1) + 400; hVideo2.Position([3 4]) = [750 500];
Initialize some variables which will be used later.
points = cornerPoints(zeros(0, 2)); features = binaryFeatures(zeros([0 64], 'uint8')); failedToMatchPoints = true; % A new mosaic will be created if % failedToMatchPoints is true
Create a processing loop to create mosaic from the input video. This loop uses the System objects you instantiated above.
while ~isDone(hsrc) % Save the points and features computed from the previous image pointsPrev = points; featuresPrev = features; % To speed up mosaicking, select and process every 5th image for i = 1:5 rgb = step(hsrc); if isDone(hsrc) break; end end % Convert the image from RGB to intensity. I = rgb2gray(rgb); % Detect corners in the image corners = detectFASTFeatures(I); % Extract FREAK feature vectors for the corners [features, points] = extractFeatures(I, corners); % Match features computed from the current and the previous images indexPairs = matchFeatures(features, featuresPrev); % Check if there are enough corresponding points in the current and the % previous images if size(indexPairs, 1) > 2 matchedPoints = points(indexPairs(:, 1), :); matchedPointsPrev = pointsPrev(indexPairs(:, 2), :); % Find corresponding locations in the current and the previous % images, and compute a geometric transformation from the % corresponding locations [tform, ~, ~, failedToMatchPoints] = estimateGeometricTransform(... matchedPoints, matchedPointsPrev, 'affine'); end if failedToMatchPoints % If the current image does not match the previous one, reset the % transformation and the mosaic image xtform = eye(3); mosaic = zeros(h, w, 3, 'single'); else % If the current image matches with the previous one, compute the % transformation for mapping the current image onto the mosaic % image xtform = xtform * tform.T; end % Display the current image and the corner points cornerImage = insertMarker(rgb, corners.Location, 'Color', 'red'); step(hVideo1, cornerImage); % Creat a mask which specifies the region of the transformed image. mask = imwarp(ones(size(I)), affine2d(xtform), 'OutputView', outputView) >= 1; % Warp the current image onto the mosaic image transformedImage = imwarp(rgb, affine2d(xtform), 'OutputView', outputView); mosaic = step(halphablender, mosaic, transformedImage, mask); step(hVideo2, mosaic); end release(hsrc);
The Corners window displays the input video along with the detected corners and Mosaic window displays the mosaic created from the input video.