| Video and Image Processing Blockset™ | |
Analysis & Enhancement
The Optical Flow block estimates the direction and speed of object motion from one image to another or from one video frame to another using either the Horn-Schunck or the Lucas-Kanade method.
| Port | Output | Supported Data Types | Complex Values Supported |
|---|---|---|---|
I/I1 | Scalar, vector, or matrix of intensity values |
| No |
I2 | Scalar, vector, or matrix of intensity values | Same as I port | No |
|V|^2 | Matrix of velocity magnitudes | Same as I port | No |
V | Matrix of velocity components in complex form | Same as I port | Yes |
To compute the optical flow between two images, you must solve the following optical flow constraint equation:
In this equation, the following values are represented:
,
and
are the spatiotemporal image
brightness derivatives
u is the horizontal optical flow
v is the vertical optical flow
Because this equation is underconstrained, there are several methods to solve for u and v:
Horn-Schunck Method
Lucas-Kanade Method
See the following two sections for descriptions of these methods
By assuming that the optical flow is smooth over the entire
image, the Horn-Schunck method computes an estimate of the velocity
field,
, that minimizes this equation:
In this equation,
and
are the spatial derivatives
of the optical velocity component u, and
scales the global smoothness
term. The Horn-Schunck method minimizes the previous equation to obtain
the velocity field, [u v], for each pixel in
the image, which is given by the following equations:
In this equation,
is the velocity
estimate for the pixel at (x,y), and
is the neighborhood average
of
. For k=0,
the initial velocity is 0.
If you set the Method parameter to Horn-Schunck, the block solves for u and v as follows:
Compute
and
using the Sobel convolution
kernel:
, and its transposed form for
each pixel in the first image.
Compute
between images
1 and 2 using the
kernel.
Assume the previous velocity to be 0, and compute
the average velocity for each pixel using
as a convolution kernel.
Iteratively solve for u and v.
Use the Compute optical flow between parameter to specify whether to compute the optical flow between two images or two video frames. If you select Current frame and N-th frame back, the N parameter appears in the dialog box. Enter a scalar value that represents the number of frames between the reference frame and the current frame.
Use the Velocity output parameter to specify
the block's output. If you select Magnitude-squared,
the block outputs the optical flow matrix where each element is of
the form
. If you select Horizontal
and vertical components in complex form, the block outputs
the optical flow matrix where each element is of the form
. The horizontal velocity component
represents the real part of each value and the vertical velocity component
represents the imaginary part of each value.
The smoothness factor,
,
is a positive constant. If the relative motion between the two images
or video frames is large, enter a large positive scalar value for
the Smoothness factor. If the relative motion
is small, enter a small positive scalar value. You must experiment
to find the smoothness factor that best suits your application.
The Optical Flow block uses an iterative process to calculate the optical flow between two images or two video frames. Use the Stop iterative solution parameter to control when the iterative process stops. If you want it to stop when the velocity difference is below a certain threshold value, select When velocity difference falls below threshold. Then, use the Velocity difference threshold parameter to specify a threshold value. If you want the iterative process to stop after a certain number of iterations, choose When maximum number of iterations is reached. Then use the Maximum number of iterations parameter to specify the maximum number of iterations you want the block to perform. If you select Whichever comes first, you must enter values for both the Velocity difference threshold and Maximum number of iterations parameters.
The block stops iterating as soon as one of these conditions is satisfied.
To solve the optical flow constraint equation for u and v,
the Lucas-Kanade method divides the original image into smaller sections
and assumes a constant velocity in each section. Then, it performs
a weighted least-square fit of the optical flow constraint equation
to a constant model for
in each section,
, by minimizing the following
equation:
Here, W is a window function that emphasizes the constraints at the center of each section. The solution to the minimization problem is given by the following equation:
If you set the Method parameter to Lucas-Kanade,
the block computes
using a difference
filter or a derivative of a Gaussian filter.
The two following sections explain how
,
,
, and then u and v are
computed.
If you set the Temporal gradient filter parameter to Difference filter [-1 1], the block solves for u and v as follows:
Compute
and
using the kernel
and its transposed form.
If you are working with fixed-point data types, the kernel values are signed fixed-point values with word length equal to 16 and fraction length equal to 15.
Compute
between images
1 and 2 using the
kernel.
Smooth the gradient components,
,
, and
, using a separable and isotropic
5-by-5 element kernel whose effective 1-D coefficients are
. If you are working with fixed-point
data types, the kernel values are unsigned fixed-point values with
word length equal to 8 and fraction length equal to 7.
Solve the 2-by-2 linear equations for each pixel using the following method:
If
Then the eigenvalues of A are
In the fixed-point diagrams,
When the block finds the eigenvalues, it compares
them to the threshold,
, that corresponds
to the value you enter for the Threshold for noise reduction parameter.
The results fall into one of the following cases:
Case 1:
and
A is nonsingular, so the block solves the system of equations using Cramer's rule.
Case 2:
and
A is singular (noninvertible), so the block normalizes the gradient flow to calculate u and v.
Case 3:
and
The optical flow, u and v, is 0.
The Compute optical flow between, N, and Velocity output parameters are described in Horn-Schunck Method.
Use the Threshold for noise reduction parameter to eliminate the effect of small movements between frames. The higher the number, the less small movements impact the optical flow calculation. Experiment with this parameter to find the value that best suits your application.
If you set the Temporal gradient filter parameter to Derivative of Gaussian, the block solves for u and v using the following steps. You can see the flow chart for this process at the end of this section:
Compute
and
using the following steps:
Use a Gaussian filter to perform temporal filtering. Specify the temporal filter characteristics such as the standard deviation and number of filter coefficients using the Number of frames to buffer for temporal smoothing parameter.
Use a Gaussian filter and the derivative of a Gaussian filter to smooth the image using spatial filtering. Specify the standard deviation and length of the image smoothing filter using the Standard deviation for image smoothing filter parameter.
Compute
between images
1 and 2 using the following steps:
Use the derivative of a Gaussian filter to perform temporal filtering. Specify the temporal filter characteristics such as the standard deviation and number of filter coefficients using the Number of frames to buffer for temporal smoothing parameter.
Use the filter described in step 1b to perform spatial filtering on the output of the temporal filter.
Smooth the gradient components,
,
, and
, using a gradient smoothing
filter. Use the Standard deviation for gradient smoothing
filter parameter to specify the standard deviation and
the number of filter coefficients for the gradient smoothing filter.
Solve the 2-by-2 linear equations for each pixel using the following method:
If
Then the eigenvalues of A are
When the block finds the eigenvalues, it compares
them to the threshold,
, that corresponds
to the value you enter for the Threshold for noise reduction parameter.
The results fall into one of the following cases:
Case 1:
and
A is nonsingular, so the block solves the system of equations using Cramer's rule.
Case 2:
and
A is singular (noninvertible), so the block normalizes the gradient flow to calculate u and v.
Case 3:
and
The optical flow, u and v, is 0.
Select the Discard normal flow estimates when constraint equation is ill-conditioned check box if you want the block to set the motion vector to zero when the optical flow constraint equation is ill-conditioned. The block calculates these motion vectors on a pixel-by-pixel basis.
Select the Output image corresponding to motion vectors (accounts for block delay) check box if you want the block to output the image that corresponds to the motion vector being output by the block.
The Velocity output parameter is described in Horn-Schunck Method.
Use the Threshold for noise reduction parameter to eliminate the effect of small movements between frames. The higher the number, the less small movements impact the optical flow calculation. Experiment with this parameter to find the value that best suits your application.
The following diagrams shows the data types used in the Optical Flow block for fixed-point signals. The block supports fixed-point data types only when the Method parameter is set to Lucas-Kanade.
You can set the product output, accumulator, gradients, threshold, and output data types in the block mask.
The Main pane of the Optical Flow dialog box appears as shown in the following figure.
Select the method the block uses to calculate the optical flow. Your choices are Horn-Schunck or Lucas-Kanade.
Select Two images to compute the optical flow between two images. Select Current frame and N-th frame back to compute the optical flow between two video frames that are N frames apart.
This parameter is visible if you set the Method parameter to Horn-Schunck or you set the Method parameter to Lucas-Kanade and the Temporal gradient filter to Difference filter [-1 1].
Enter a scalar value that represents the number of frames between the reference frame and the current frame. This parameter becomes available if you set the Compute optical flow between parameter, you select Current frame and N-th frame back.
If the relative motion between the two images or video frames is large, enter a large positive scalar value. If the relative motion is small, enter a small positive scalar value. This parameter becomes available if you set the Method parameter to Horn-Schunck.
Use this parameter to control when the block's iterative solution process stops. If you want it to stop when the velocity difference is below a certain threshold value, select When velocity difference falls below threshold. If you want it to stop after a certain number of iterations, choose When maximum number of iterations is reached. You can also select Whichever comes first. This parameter becomes available if you set the Method parameter to Horn-Schunck.
Enter a scalar value that represents the maximum number of iterations you want the block to perform. This parameter is only visible if, for the Stop iterative solution parameter, you select When maximum number of iterations is reached or Whichever comes first. This parameter becomes available if you set the Method parameter to Horn-Schunck.
Enter a scalar threshold value. This parameter is only visible if, for the Stop iterative solution parameter, you select When velocity difference falls below threshold or Whichever comes first. This parameter becomes available if you set the Method parameter to Horn-Schunck.
If you select Magnitude-squared,
the block outputs the optical flow matrix where each element is of
the form
. If you select Horizontal
and vertical components in complex form, the block outputs
the optical flow matrix where each element is of the form
.
Specify whether the block solves for u and v using a difference filter or a derivative of a Gaussian filter. This parameter becomes available if you set the Method parameter to Lucas-Kanade.
Use this parameter to specify the temporal filter characteristics such as the standard deviation and number of filter coefficients. This parameter becomes available if you set the Temporal gradient filter parameter to Derivative of Gaussian.
Specify the standard deviation for the image smoothing filter. This parameter becomes available if you set the Temporal gradient filter parameter to Derivative of Gaussian.
Specify the standard deviation for the gradient smoothing filter. This parameter becomes available if you set the Temporal gradient filter parameter to Derivative of Gaussian.
Select this check box if you want the block to set the motion vector to zero when the optical flow constraint equation is ill-conditioned. This parameter becomes available if you set the Temporal gradient filter parameter to Derivative of Gaussian.
Select this check box if you want the block to output the image that corresponds to the motion vector being output by the block. This parameter becomes available if you set the Temporal gradient filter parameter to Derivative of Gaussian.
Enter a scalar value that determines the motion threshold between each image or video frame. The higher the number, the less small movements impact the optical flow calculation. This parameter becomes available if you set the Method parameter to Lucas-Kanade.
The Fixed-point pane of the Optical Flow dialog box appears as shown in the following figure.
Select the rounding mode for fixed-point operations.
Select the overflow mode for fixed-point operations.
Use this parameter to specify how to designate the product output word and fraction lengths.
When you select Binary point scaling, you can enter the word length and the fraction length of the product output in bits.
When you select Slope and bias scaling, you can enter the word length in bits and the slope of the product output. The bias of all signals in the Video and Image Processing Blockset blocks is 0.
Use this parameter to specify how to designate this accumulator word and fraction lengths.
When you select Same as product output, these characteristics match those of the product output.
When you select Binary point scaling, you can enter the word length and the fraction length of the accumulator in bits.
When you select Slope and bias scaling, you can enter the word length in bits and the slope of the accumulator. The bias of all signals in the Video and Image Processing Blockset blocks is 0.
Choose how to specify the word length and fraction length of the gradients data type:
When you select Same as accumulator, these characteristics match those of the accumulator.
When you select Same as product output, these characteristics match those of the product output.
When you select Binary point scaling, you can enter the word length and the fraction length of the quotient, in bits.
When you select Slope and bias scaling, you can enter the word length in bits and the slope of the quotient. The bias of all signals in the Video and Image Processing Blockset blocks is 0.
Choose how to specify the word length and fraction length of the threshold data type:
When you select Same word length as first input, the threshold word length matches that of the first input.
When you select Specify word length, enter the word length of the threshold data type.
When you select Binary point scaling, you can enter the word length and the fraction length of the threshold, in bits.
When you select Slope and bias scaling, you can enter the word length in bits and the slope of the threshold. The bias of all signals in the Video and Image Processing Blockset blocks is 0.
Choose how to specify the word length and fraction length of the output data type:
When you select Binary point scaling, you can enter the word length and the fraction length of the output, in bits.
When you select Slope and bias scaling, you can enter the word length in bits and the slope of the output. The bias of all signals in the Video and Image Processing Blockset blocks is 0.
Select this parameter to prevent any fixed-point scaling you specify in this block mask from being overridden by the autoscaling tool in the Fixed-Point Tool. For more information, see fxptdlg, a reference page on the Fixed-Point Tool in the Simulink documentation.
[1] Barron, J.L., D.J. Fleet, S.S. Beauchemin, and T.A. Burkitt. Performance of optical flow techniques. CVPR, 1992.
Video and Image Processing Blockset software | |
Video and Image Processing Blockset software |
| Opening | Projective Transformation | |
| © 1984-2008- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |