This code implements two variations of the paper
"High accuracy optic flow using a theory for warping" presented at ECCV 2004 by Dr. Thomas Brox.
Its variants appeared in "Particle Video"
presented at CVPR 2006 by Dr. Peter Sand.
The files optical_flow_brox.m and
optical_flow_sand.m are the files you need to run.
optical_flow_brox.m implements the traditional algorithm. Just replace the first lines where images are read, with your image, and start the process.
optical_flow_sand.m implements a variant presented in CVPR 2006. Again just replace the lines "img1 = imread('...')" and "img2 = imread('...')" with your images and the rest is automatic.
The variable num_levels controls the number of images in the laplacian pyramid to be constructed. Change it to suit your convenience.
The way to tweak / use the code is described in the documentation pdf that is attached with the code. It can also be accessed from
In order to tweak the code, however, you need to read the two papers...
While using the code I faced a problem. I didn't change anything. What can I do?
Thanks in advance.
Error using conv2
HCOL, HROW, and A must be single or double.
Error in gaussianSmooth (line 46)
img_smooth = conv2(grid, grid, img, 'same') ;
Error in gaussianRescaling (line 22)
img_smooth = gaussianSmooth(img, 1.0/scale_factor, 1e-3) ;
Error in optic_flow_brox (line 18)
im1_hr = gaussianRescaling( img1, power( 0.95, num_levels ) ) ; % Rescaling images
Error in brox_flow (line 4)
Generally a good implementation.
I had to change the indices in the iteration in optic_flow_brox to count to 0 and not to 1, otherwise the function crashed since the size of u and v where different to the size of the input images.
Took me hours to find the bug.
Maybe this helps others.
I am trying to use the optic_flow_sand function and encountering multiple issues.
* The rescale factor between levels should be a variable, not the constant 0.9.
* The function crashes as the size of u and v is not equal to the final image size.
* The function only works for RGB images, which is painful when I use gray images.
Sorry if I misunderstood your code.
I am wondering if the code you submitted assumes "non-linearity" in the data term or the "linearity" of the term because as
I briefed through your code and I couldn't find a place where you are doing the two fixed point iterations, one to get rid of non-linear flow equation and the other is to get rid of the non-linear data term before solving for the flow increments using SOR.
Correct me if I am wrong!!!
Just I was wondering if this code could work for a grays-cale images, or there is another version of the code. what I am doing now is to convert the gray images to rgb and then estimate motion. I want to save more time by doing it directly on the gray images if possible, thanks for the help.
Its a really nice piece of work, I am getting stuck at one part because, I want to reduce the number of points/resolution for which optical flow is computed,I believe that this will make the code faster.
Would be grateful,if anyone who has done this provides me with some help.
Hi all,I have a question.I compute the optical flow(u ,v) using LK and HS method. Now I want to compute the average angular error(AAE).But I don't know how.Do I need the ground truth optical flow (in .pcm file format)for compare the flow ?
Or any other way to compute the angular error?
thank u all.
I have made some minor modifications to your implementation of Brox's algorithm:
- Reflecting boundary conditions corresponds to mirror-extending the images. This can remove some of "unwanted" spurious flows at image boundaries.
- Added color channels (and different color spaces)
- 500 SOR Iterations is too much! If I remember correctly, Brox mentions much less SOR iterations. You can safely decrease it to around 50(or even less).
Let me know if you want me to send the modified code to you(Visesh).
Regarding the higher errors on the results:
- If I'm not mistaken, this implementation does not assume spatio-temporal smoothness(it's not 3D) and does not work on more than 2 images . In order to get nicer results you have to extend it to the 3D case.
I recently realized that Brox has already put his binaries(both for the ECCV04 and CVPR09 papers) online!
Thank you for this implementation. It is valuable in an educational way. But given that the bugs mentioned above are still in this package this gets only 3 stars from me. Also, you might be interested in this implementation which is a lot faster and produces very similar results:
Very usefull tool ! I'm looking forward to a MEX version.
I have the code, but its not documented yet... I could give it to you if you want to have a look, but I haven't thoroughly proofread it for bugs, and might not have time before december....
Visesh, you mentioned a MEX version of the algorithm. Have you made it available somewhere? I'm trying to run it on rather large images, so I could appreciate some improved speed. Thanks.
Hi. First of all thanks for your share.
I am trying to understand your code and I am almost finished but 'constructMatrix' function is still waiting to be explored. Could you, if possible, give some explanations on construction of the matrices in this script.
Great work. Thanks for sharing. Haven't gone through the documentation yet, but the preset configurations worked reasonably well.
Really great work!
I am using this code for few of my images. I am not able to get the (u, v) of the same dimension as that of input images. I am getting (u,v) equal to the next resolution in the pyramid dependng on the scale_factor I give. For Instance, If I am giving two 120*120 images, scale_factor = 0.5, no. of levels = 3, I am getting final output (u, v) of the resolution 60*60. How to obtain (u,v) in the resolution of input images. Am I wrong in specifying some parameters?
Great Piece of Work !!!!
Great work!! Hi Vishesh, When I run the simulation of brox, in the lower left corner of quiver plot, there is some additional unwanted vector fields, i want to cope with them, so do you have any idea how to cope and why the are occuring?
Great! do you have the implementation in c?
Thanks for sharing. very well documented and extermely useful.
can you tell me what doesnt work ? i have received no complains so far... may be you did not read the reference document i sent... if possible post the error so that i can check
Brox implementation does not work. There is something wrong with it..
Thank you, i am waiting. can't i resize u and v a little bit to be bigger? in the end, the original image is 148*148 and the im2_hr is a little bit less, so i want to resize the u and v and then warp the original image?
Hi, the basic smoothing operation that is done in the algorithm is to reduce the effects of noise in the data, so it cannot be avoided. If you really have noiseless data, and do not need much smoothing, you can change the appropriate parameters. You can refer to my documentation, but will eventually have to tweak the code to reduce the variance of the gaussian used for smoothing. If this is not too urgent (u can wait a week or so), I should be able to post additional documentation material, which you can use...
Hi, i have a question: the registration is GREAT, but aventully i get an image that is not as "focused" as the original image, it looks like filtered with gaussian filter... so, the question is how can i obtain aventually the original imag warped, and not the im2_hr ? thank you very much
it should work, so long as u and v are formatted like those in my code, which is just two 2D arrays with flow values....
Hello i have a question, can i use the mywarp() function for any U,V i have that wasn't comuted by the algorithm? i mean, U,V are calculated not through this algorithm, but i want to use the warp function, will it work? or this is specially designed for this algorithm?
It is a very usefull demo as a state-of-the-art optical flows method. Thanks for sharing it!
Thanks for your comments. I am sorry to have posted a wrong version of the code, hence the wrong limits for inner and outer iterations. I did not notice that while uploading. Thanks for pointing it out. I will upload the corrected version in some time.
You are right about the parameters. They are very important, and have to be carefully tuned to get the desired results, as per the author.
The algorithm I have coded differs from the original formulation in some of the boundary conditions and smoothing operations.. (will post a documentation about it in the next 2 months). This is possibly the reason why the figures reported in the literature are not achieved by this code. Otherwise the code follows the algorithm faithfully, to my knowledge.
Also, do watch this space for a MEX version of the above algorithm. Its faster than the current version, with exactly the same results.
Although there are a couple of bugs in it, it's a good implementation anyway. You need fix the bugs when you are trying to use it. The two parameters,the smoothness one and the gradient one, are very important.When I did an experiment for face tracking and I set the smoothness weight to a big value or a small value, the error is unbearable.But finally I got a reasonable result when the parameter of this one and the gradient one are set suitable for my sequence.
on Yosemite I get the Sand&Teller version to an angular error of AAE=3.4 degrees and endpoint error of AEP=0.17 pixels, excluding a 10 pixel border (like most evaluations do to limit mean boundary effects). This is after fixing the bug mentioned in your posting, and setting parameters
outer_iter = 20; inner_iter = 20;
mu = 0.95;
% mu ... pyramid scale factor, hardcoded in original source
The errors are not quite as low as those reported by Brox, but competitive with other recent entries in the Middlebury flow evaluation.
First of all, it is a great relief that finally someone published an implementation to some state-of-the-art optical flows.
However, I am a little bit curious about your implementation.
First there are some bugs in it.
In the Sand-Version your most outer loop does not rescale your solution to the initial resolution of the input images, as your loop variable stops at 8 but should go to 1.
A similar bug is in the brox version, where i stops at 1 but should stop at 0.
I tried your implementation on the yosemite sequence but I got horrible results. Using the sand-version I got an angular error of 12 degree and using the brox-version I got an angular error of 20, which is one magnitude larger than it should be!
I would be interested in the explanation for this, because it is rather strange.
Did you already make tests with the yosemite sequence by yourself? Should one choose other parameters maybe? If so, what parameters work well for this sequence.
Without comparable angular errors it is hard to decide, whether your implementation is correct.
I am using Matlab 22.214.171.12401 (R14). In this
case the line of code I wrote and the one
you have written given the same results.
Is there something I am not following ? Is there a difference for different versions ? I do not have access to other matlab versions so I cannot be sure. Please let me know if I am missing something...
I think there is a bug line in gaussianRescaling line 31.
img_scaled = imresize(img_smooth, scale_factor, 'bilinear', 0);
img_scaled = imresize(img_smooth, [fix(size(img_smooth,1)*scale_factor) fix(size(img_smooth,2)*scale_factor)], 'bilinear', 0);
I am to some extent friendly with different optical flow and warping techniques and I think that this code is worth a PhD project.
The algorithm makes use of high mathematical background.
I myself could understand the flow of the codes in less than an hour without reading help!
Ofcourse I do not recommend this code for those who are novice in the field of motion estimation and registration while the algorithm relies on newly developed penalizations and warping strategies.
Thanks for sharing this, Visesh!
Five stars from me!!
Hi Mr Moisy,
Thank you very much for your comments. Have incorporated all of them in the current version.
OK, I have seen the (little) help of the revised version of this submission, thanks ; however I think it should still be improved to be really useful to other users:
a) changing the filenames directly in the script is awful: you can't automatize the processing for several files! Just make it a function, in the form M = optic_flow(IM1, IM2)
b) It still does not work for me. optic_flow_sand returns the error message "??? Undefined function or method 'getalphaImg' for input arguments of type 'uint8'." and optic_flow_brox returns "??? Undefined function or method 'split' for input arguments of type 'double'." (I use Matlab 7.5).
c) I understand that for copyright reasons you cannot include the two refered papers into the submission; however you should explain a little bit what does each algorithm.
How to get started? Please provide some help and examples! I was not present at the ECCV 2004 conference so I don't know how to start with those 20 files. As it stands, this submission is useless.
Updated with license removed
Added a some documentation.
Updating information on how to get started.
I am adding a screenshot of the results of the current algorithm/code.