Average Optical Flow vectors and plot over multiple frames

Hello,

I am trying to get the average optical flow vectors for a whole video (e.g. for a 10 second video). Currently I am using the opticalFlowFarneback method from Estimate optical flow example with the Computer Vision Toolbox (https://www.mathworks.com/help/vision/ref/opticalflowhs.estimateflow.html):

%% Estimating Optical Flow
% This example uses the Farneback Method to to estimate the direction and speed of moving
% cars in the video
% Copyright 2018 The MathWorks, Inc.
%% Read the video into MATLAB
vidReader = VideoReader('visiontraffic.avi');
opticFlow = opticalFlowFarneback;
%% Estimate Optical Flow of each frame
while hasFrame(vidReader)
frameRGB = readFrame(vidReader);
frameGray = rgb2gray(frameRGB);
flow = estimateFlow(opticFlow,frameGray);
imshow(frameRGB)
hold on
% Plot the flow vectors
plot(flow,'DecimationFactor',[25 25],'ScaleFactor', 2)
% Find the handle to the quiver object
q = findobj(gca,'type','Quiver');
% Change the color of the arrows to red
q.Color = 'r';
drawnow
hold off
end

Although this produces a nice visualization of flow vectors between two frames (see attached image), I need more like an average of flow vectors across all frames of the video. I am quite new to MatLab, I would be super thankful for any feedback how to change abovementioned code.

[Adam Danz]

Your question is framed in such a way that only people familiar with the tutorial you mentioned can help.

If you can redefine the question more generally or provide a minimal working example, you might have a better chance of a bite.

[Alex M]

thank you for your comment! You are right, I edited my initial post so I hope my question might be easier to understand now.

Answer 1

[Adam Danz]

Here's a demo that shows how to

• Store the flow structure within a loop (assumes all flow arrays are the same size)
• Average a variable in the flow structure across loop iterations.

% preallocate structure array
flow = opticalFlow;

% Create and store optic flow object within loop
for i = 1:10
    Vx = randn(100,100);
    Vy = randn(100,100);
    flow(i) = opticalFlow(Vx,Vy);
end
% flow is a 1x10 structure array

% Concatenate Orientation along 3rd dimension
% flowOrientation is a 100x100x10 array for 
% 100x100 flow vectors across 10 iterations.
flowOrientation = cat(3,flow.Orientation);

% Average Orientation values across loop iterations
% meanOrientation is 100x100 mean vectors
meanOrientation = mean(flowOrientation,3)

[Alex M]

Thank you for you Answer :) I am sorry for keep asking, as I said I am really new to Matlab. After I saved the flow in every iteration in f the result is 1x308 cell. In that cell, 308 optical flow values are stored, containing each the properties Vx, Vy, Orientation and Magnitude.

Is there a simple way to average these properties and hand them over to the plot function after the loop? That would be super helpful. Thank you in advance!

[Adam Danz]

I looked into the documentation and saw that flow is a structure.

It's better to store the data in a structure array rather than a cell array. I've updated my answer to demonstrate the workflow.

[Alex M]

Thanks again for the reply, this was very helpful! This is hopefully my last question. As you recommended, I stored the flow in a structure within the while loop:

while hasFrame(vidReader)
frameRGB = readFrame(vidReader);
frameGray = rgb2gray(frameRGB);
flow(j) = estimateFlow(opticFlow,frameGray);
imshow(frameRGB);
hold on
% Plot the flow vectors
plot(flow(j),'DecimationFactor',[25 25],'ScaleFactor', 5)
drawnow
hold off
j = j + 1;
end

I used very simple and short hamburg taxi sequence as quick benchmark. flow is in my case a 1x41 optical flow array. After the loop, I then concatenate Vx and Vy from flow along the 3rd dimension. Afterwards I create a new optical flow object called flower and hand it over to the plot function.

flowVx = cat(3,flow.Vx);
flowVy = cat(3,flow.Vy);
meanVx = mean(flowVx,3);
meanVy = mean(flowVy,3);
flower = opticalFlow(meanVx,meanVy);
plot(flower,'DecimationFactor',[25 25],'ScaleFactor', 10)
q = findobj(gca,'type','Quiver');
q.Color = 'r';

However the result seems a bit weird. Although the resulting vectors seem reasonable in terms of direction, I would expect them on the lower part on the plot instead of being in the upper part. What I am still doing wrong? Help!

[Adam Danz]

My guess is that you're plotting the image with imshow or some other image plotting function. These functions plot images with reversed y-axis. You can turn on the axes of an image plot to confirm that.

So the strong rightward vectors around y = 125 is actually toward the bottom of the image right around the y location of the dark car moving rightward.

If you want to overlay the optic flow vectors on top of the image you can flip the matrix vertically.

[Alex M]

Thanks! I think that did the trick! The result after flipping the y-axis looks very reasonable to me. Although I was wondering why this was not issue with the example Optical Flow code in the original Post. Thank you very much for your support, it helped me a lot!

[Adam Danz]

I see that you're using 2020b and that example in 2021a looks like it changed. I'd have to look at the documentation for 2020b which I don't have time to do right now but maybe I'll remember to look into it later.