Track points in video using KanadeLucasTomasi (KLT) algorithm
The point tracker object tracks a set of points using the KanadeLucasTomasi (KLT), featuretracking algorithm. You can use the point tracker for video stabilization, camera motion estimation, and object tracking. It works particularly well for tracking objects that do not change shape and for those that exhibit visual texture. The point tracker is often used for shortterm tracking as part of a larger tracking framework.
As the point tracker algorithm progresses over time, points can be lost due to lighting variation, out of plane rotation, or articulated motion. To track an object over a long period of time, you may need to reacquire points periodically.
Starting in R2016b, instead of using the step
method
to perform the operation defined by the System
object™, you can
call the object with arguments, as if it were a function. For example, y
= step(obj,x)
and y = obj(x)
perform
equivalent operations.
pointTracker = vision.PointTracker
returns
a System object, pointTracker
, that tracks a set
of points in a video.
pointTracker = vision.PointTracker(
configures
the tracker object properties, specified as one or more namevalue
pair arguments. Unspecified properties have default values. Name,Value
)
To initialize the tracking process, you must use the initialize
method
to specify the initial locations of the points and the initial video
frame.
initialize(pointTracker,points,I)
initializes
points to track and sets the initial video frame. The initial locations POINTS
,
must be an Mby2 array of [x y] coordinates. The
initial video frame, I
, must be a 2D grayscale
or RGB image and must be the same size and data type as the video
frames passed to the step
method.
The detectFASTFeatures
, detectSURFFeatures
, detectHarrisFeatures
, and detectMinEigenFeatures
functions are
few of the many ways to obtain the initial points for tracking.
Define and set up your point tracker object using the constructor.
Call the step
method with the input
image, I
, and the point tracker object, pointTracker
.
See the following syntax for using the step
method.
After initializing the tracking process, use the step
method
to track the points in subsequent video frames. You can also reset
the points at any time by using the setPoints
method.
[points,point_validity] = step(pointTracker,I)
tracks
the points in the input frame, I
using the point
tracker object, pointTracker
. The output points
contain
an Mby2 array of [x y] coordinates that correspond
to the new locations of the points in the input frame, I
.
The output, point_validity
provides an Mby1
logical array, indicating whether or not each point has been reliably
tracked.
A point can be invalid for several reasons. The point can become
invalid if it falls outside of the image. Also, it can become invalid
if the spatial gradient matrix computed in its neighborhood is singular.
If the bidirectional error is greater than the MaxBidirectionalError
threshold,
this condition can also make the point invalid.
[points,point_validity,scores] = step(pointTracker,I)
additionally
returns the confidence score for each point. The Mby1
output array, scores
, contains values between 0
and 1
.
These values correspond to the degree of similarity between the neighborhood
around the previous location and new location of each point. These
values are computed as a function of the sum of squared differences
between the previous and new neighborhoods. The greatest tracking
confidence corresponds to a perfect match score of 1
.
setPoints(pointTracker, points)
sets the
points for tracking. The method sets the Mby2 points
array
of [x y] coordinates with the
points to track. You can use this method if the points need to be
redetected because too many of them have been lost during tracking.
setPoints(pointTracker,points,point_validity)
additionally
lets you mark points as either valid or invalid. The input logical
vector point_validity
of length M,
contains the true or false value corresponding to the validity of
the point to be tracked. The length M corresponds
to the number of points. A false value indicates an invalid point
that should not be tracked. For example, you can use this method with
the estimateGeometricTransform
function
to determine the transformation between the point locations in the
previous and current frames. You can mark the outliers as invalid.

Number of pyramid levels Specify an integer scalar number of pyramid levels. The point
tracker implementation of the KLT algorithm uses image pyramids. The
object generates an image pyramid, where each level is reduced in
resolution by a factor of two compared to the previous level. Selecting
a pyramid level greater than 1, enables the algorithm to track the
points at multiple levels of resolution, starting at the lowest level.
Increasing the number of pyramid levels allows the algorithm to handle
larger displacements of points between frames. However, computation
cost also increases. Recommended values are between
Each pyramid level is formed by downsampling the previous level by a factor of two in width and height. The point tracker begins tracking each point in the lowest resolution level, and continues tracking until convergence. The object propagates the result of that level to the next level as the initial guess of the point locations. In this way, the tracking is refined with each level, up to the original image. Using the pyramid levels allows the point tracker to handle large pixel motions, which can comprise distances greater than the neighborhood size. Default: 

Forwardbackward error threshold Specify a numeric scalar for the maximum bidirectional error.
If the value is less than Using the bidirectional error is an effective way to eliminate
points that could not be reliably tracked. However, the bidirectional
error requires additional computation. When you set the Default: 

Size of neighborhood Specify a twoelement vector, [height, width] to represent the
neighborhood around each point being tracked. The height and width
must be odd integers. This neighborhood defines the area for the spatial
gradient matrix computation. The minimum value for Default: 

Maximum number of search iterations Specify a positive integer scalar for the maximum number of
search iterations for each point. The KLT algorithm performs an iterative
search for the new location of each point until convergence. Typically,
the algorithm converges within 10 iterations. This property sets the
limit on the number of search iterations. Recommended values are between Default: 
initialize  Initialize video frame and points to track 
setPoints  Set points to track 
step  Track points in video using KanadeLucasTomasi (KLT) algorithm 
Common to All System Objects  

clone  Create System object with same property values 
getNumInputs  Expected number of inputs to a System object 
getNumOutputs  Expected number of outputs of a System object 
isLocked  Check locked states of a System object (logical) 
release  Allow System object property value changes 
Lucas, Bruce D. and Takeo Kanade. “An Iterative Image Registration Technique with an Application to Stereo Vision,”Proceedings of the 7th International Joint Conference on Artificial Intelligence, April, 1981, pp. 674–679.
Tomasi, Carlo and Takeo Kanade. Detection and Tracking of Point Features, Computer Science Department, Carnegie Mellon University, April, 1991.
Shi, Jianbo and Carlo Tomasi. “Good Features to Track,” IEEE Conference on Computer Vision and Pattern Recognition, 1994, pp. 593–600.
Kalal, Zdenek, Krystian Mikolajczyk, and Jiri Matas. “ForwardBackward Error: Automatic Detection of Tracking Failures,” Proceedings of the 20th International Conference on Pattern Recognition, 2010, pages 2756–2759, 2010.
detectHarrisFeatures
 detectMinEigenFeatures
 detectSURFFeatures
 estimateGeometricTransform
 imrect
 insertMarker
 vision.HistogramBasedTracker