High accuracy optical flow

High accuracy optical flow using a theory for warping

Description

This code implements two variations of the paper
"High accuracy optic flow using a theory for warping" presented at ECCV 2004 by Dr. Thomas Brox.

Its variants appeared in "Particle Video"
presented at CVPR 2006 by Dr. Peter Sand.

The files optical_flow_brox.m and
optical_flow_sand.m are the files you need to run.

optical_flow_brox.m implements the traditional algorithm. Just replace the first lines where images are read, with your image, and start the process.

optical_flow_sand.m implements a variant presented in CVPR 2006. Again just replace the lines "img1 = imread('...')" and "img2 = imread('...')" with your images and the rest is automatic.

The variable num_levels controls the number of images in the laplacian pyramid to be constructed. Change it to suit your convenience.

The way to tweak / use the code is described in the documentation pdf that is attached with the code. It can also be accessed from

http://perception.inrialpes.fr/~chari/myweb/Software/

In order to tweak the code, however, you need to read the two papers...

• Required Products: Image Processing Toolbox
• MATLAB release: MATLAB 7 (R14)

[F Moisy]

How to get started? Please provide some help and examples! I was not present at the ECCV 2004 conference so I don't know how to start with those 20 files. As it stands, this submission is useless.

[F Moisy]

OK, I have seen the (little) help of the revised version of this submission, thanks ; however I think it should still be improved to be really useful to other users:

a) changing the filenames directly in the script is awful: you can't automatize the processing for several files! Just make it a function, in the form M = optic_flow(IM1, IM2)

b) It still does not work for me. optic_flow_sand returns the error message "??? Undefined function or method 'getalphaImg' for input arguments of type 'uint8'." and optic_flow_brox returns "??? Undefined function or method 'split' for input arguments of type 'double'." (I use Matlab 7.5).

c) I understand that for copyright reasons you cannot include the two refered papers into the submission; however you should explain a little bit what does each algorithm.

F.

[visesh chari]

Hi Mr Moisy,

Thank you very much for your comments. Have incorporated all of them in the current version.

V.

[mathild L.]

I am to some extent friendly with different optical flow and warping techniques and I think that this code is worth a PhD project.
The algorithm makes use of high mathematical background.
I myself could understand the flow of the codes in less than an hour without reading help!
Ofcourse I do not recommend this code for those who are novice in the field of motion estimation and registration while the algorithm relies on newly developed penalizations and warping strategies.

Thanks for sharing this, Visesh!
Five stars from me!!

[Samuel Foucher]

I think there is a bug line in gaussianRescaling line 31.

This line:

img_scaled = imresize(img_smooth, scale_factor, 'bilinear', 0);

should be:

img_scaled = imresize(img_smooth, [fix(size(img_smooth,1)*scale_factor) fix(size(img_smooth,2)*scale_factor)], 'bilinear', 0);

[Visesh Chari]

Hi Samuel,

I am using Matlab 7.0.0.19901 (R14). In this
case the line of code I wrote and the one
you have written given the same results.

Is there something I am not following ? Is there a difference for different versions ? I do not have access to other matlab versions so I cannot be sure. Please let me know if I am missing something...

Thanks,
Visesh

[Christian B.]

First of all, it is a great relief that finally someone published an implementation to some state-of-the-art optical flows.

However, I am a little bit curious about your implementation.

First there are some bugs in it.
In the Sand-Version your most outer loop does not rescale your solution to the initial resolution of the input images, as your loop variable stops at 8 but should go to 1.

A similar bug is in the brox version, where i stops at 1 but should stop at 0.

I tried your implementation on the yosemite sequence but I got horrible results. Using the sand-version I got an angular error of 12 degree and using the brox-version I got an angular error of 20, which is one magnitude larger than it should be!
I would be interested in the explanation for this, because it is rather strange.
Did you already make tests with the yosemite sequence by yourself? Should one choose other parameters maybe? If so, what parameters work well for this sequence.
Without comparable angular errors it is hard to decide, whether your implementation is correct.

[Konrad S.]

Christian,

on Yosemite I get the Sand&Teller version to an angular error of AAE=3.4 degrees and endpoint error of AEP=0.17 pixels, excluding a 10 pixel border (like most evaluations do to limit mean boundary effects). This is after fixing the bug mentioned in your posting, and setting parameters

alpha_global=5; alpha_local=10;
outer_iter = 20; inner_iter = 20;
mu = 0.95;
% mu ... pyramid scale factor, hardcoded in original source

The errors are not quite as low as those reported by Brox, but competitive with other recent entries in the Middlebury flow evaluation.

cheers,
Konrad

[Hui Fang]

Although there are a couple of bugs in it, it's a good implementation anyway. You need fix the bugs when you are trying to use it. The two parameters,the smoothness one and the gradient one, are very important.When I did an experiment for face tracking and I set the smoothness weight to a big value or a small value, the error is unbearable.But finally I got a reasonable result when the parameter of this one and the gradient one are set suitable for my sequence.

[visesh chari]

Hi all,
Thanks for your comments. I am sorry to have posted a wrong version of the code, hence the wrong limits for inner and outer iterations. I did not notice that while uploading. Thanks for pointing it out. I will upload the corrected version in some time.

You are right about the parameters. They are very important, and have to be carefully tuned to get the desired results, as per the author.

The algorithm I have coded differs from the original formulation in some of the boundary conditions and smoothing operations.. (will post a documentation about it in the next 2 months). This is possibly the reason why the figures reported in the literature are not achieved by this code. Otherwise the code follows the algorithm faithfully, to my knowledge.

Also, do watch this space for a MEX version of the above algorithm. Its faster than the current version, with exactly the same results.

[shifang xu]

It is a very usefull demo as a state-of-the-art optical flows method. Thanks for sharing it!

[Assaf cohen]

Hello i have a question, can i use the mywarp() function for any U,V i have that wasn't comuted by the algorithm? i mean, U,V are calculated not through this algorithm, but i want to use the warp function, will it work? or this is specially designed for this algorithm?

[Visesh Chari]

it should work, so long as u and v are formatted like those in my code, which is just two 2D arrays with flow values....

[Assaf cohen]

Hi, i have a question: the registration is GREAT, but aventully i get an image that is not as "focused" as the original image, it looks like filtered with gaussian filter... so, the question is how can i obtain aventually the original imag warped, and not the im2_hr ? thank you very much

[Visesh Chari]

Hi, the basic smoothing operation that is done in the algorithm is to reduce the effects of noise in the data, so it cannot be avoided. If you really have noiseless data, and do not need much smoothing, you can change the appropriate parameters. You can refer to my documentation, but will eventually have to tweak the code to reduce the variance of the gaussian used for smoothing. If this is not too urgent (u can wait a week or so), I should be able to post additional documentation material, which you can use...

[Assaf cohen]

Thank you, i am waiting. can't i resize u and v a little bit to be bigger? in the end, the original image is 148*148 and the im2_hr is a little bit less, so i want to resize the u and v and then warp the original image?

[UMD Mishra]

Brox implementation does not work. There is something wrong with it..

[Visesh Chari]

can you tell me what doesnt work ? i have received no complains so far... may be you did not read the reference document i sent... if possible post the error so that i can check

[Mo Star]

Thanks for sharing. very well documented and extermely useful.

[Assaf laufer]

Great! do you have the implementation in c?

[Ayaz Ahmed Shaikh]

Great work!! Hi Vishesh, When I run the simulation of brox, in the lower left corner of quiver plot, there is some additional unwanted vector fields, i want to cope with them, so do you have any idea how to cope and why the are occuring?

[De]

Great Piece of Work !!!!

[Naveen]

Really great work!
I am using this code for few of my images. I am not able to get the (u, v) of the same dimension as that of input images. I am getting (u,v) equal to the next resolution in the pyramid dependng on the scale_factor I give. For Instance, If I am giving two 120*120 images, scale_factor = 0.5, no. of levels = 3, I am getting final output (u, v) of the resolution 60*60. How to obtain (u,v) in the resolution of input images. Am I wrong in specifying some parameters?

[Omid Aghazadeh]

Great work. Thanks for sharing. Haven't gone through the documentation yet, but the preset configurations worked reasonably well.

[tlg]

Hi. First of all thanks for your share.
I am trying to understand your code and I am almost finished but 'constructMatrix' function is still waiting to be explored. Could you, if possible, give some explanations on construction of the matrices in this script.

[tlg]

 

[Johan]

Visesh, you mentioned a MEX version of the algorithm. Have you made it available somewhere? I'm trying to run it on rather large images, so I could appreciate some improved speed. Thanks.

[Visesh Chari]

Hi Johan,

I have the code, but its not documented yet... I could give it to you if you want to have a look, but I haven't thoroughly proofread it for bugs, and might not have time before december....

[Hu Yu]

Very usefull tool ! I'm looking forward to a MEX version.

[Brian]

Thank you for this implementation. It is valuable in an educational way. But given that the bugs mentioned above are still in this package this gets only 3 stars from me. Also, you might be interested in this implementation which is a lot faster and produces very similar results:
http://people.csail.mit.edu/celiu/OpticalFlow/

[Omid Aghazadeh]

I have made some minor modifications to your implementation of Brox's algorithm:
- Reflecting boundary conditions corresponds to mirror-extending the images. This can remove some of "unwanted" spurious flows at image boundaries.
- Added color channels (and different color spaces)
- 500 SOR Iterations is too much! If I remember correctly, Brox mentions much less SOR iterations. You can safely decrease it to around 50(or even less).

Let me know if you want me to send the modified code to you(Visesh).

Regarding the higher errors on the results:
- If I'm not mistaken, this implementation does not assume spatio-temporal smoothness(it's not 3D) and does not work on more than 2 images . In order to get nicer results you have to extend it to the 3D case.

I recently realized that Brox has already put his binaries(both for the ECCV04 and CVPR09 papers) online!

[rh_sh]

Hi all,I have a question.I compute the optical flow(u ,v) using LK and HS method. Now I want to compute the average angular error(AAE).But I don't know how.Do I need the ground truth optical flow (in .pcm file format)for compare the flow ?
Or any other way to compute the angular error?
thank u all.

[shreyas]

Hey,
Its a really nice piece of work, I am getting stuck at one part because, I want to reduce the number of points/resolution for which optical flow is computed,I believe that this will make the code faster.
Would be grateful,if anyone who has done this provides me with some help.

[Kassem Al Ismaeil]

Hey,
Just I was wondering if this code could work for a grays-cale images, or there is another version of the code. what I am doing now is to convert the gray images to rgb and then estimate motion. I want to save more time by doing it directly on the gray images if possible, thanks for the help.