Thread Subject: Motion estimation vs. background subtraction

I am conducting a project which consists of object
tracking. There I want to have a good segmentation of the
moving objects from the background. The background may have
noise-factors like light changes, and wind in trees.

What kind of segmentation is best? Should I use just
background subtraction, or is it best with motion
estimation?

Doeas anybody have some thoughts on this subject?

Thanks, Helmer.

"Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
message <fr8o5t$m76$1@fred.mathworks.com>...
> I am conducting a project which consists of object
> tracking. There I want to have a good segmentation of the
> moving objects from the background. The background may
have
> noise-factors like light changes, and wind in trees.
>
> What kind of segmentation is best? Should I use just
> background subtraction, or is it best with motion
> estimation?
>
> Doeas anybody have some thoughts on this subject?
>
> Thanks, Helmer.

1) Use a colour camera

2) Convert to Normalized RGB to remove the effect of
illumination variation.

3) Over a protracted period, with no moving target in the
scene, calculate the mean background intensity of each
individual pixel, together with its associated standard
deviation.

4) Select a confidence level (e.g. 3xsigma) whereby the
difference between the mean image and the current image is
deemed to be statistically similar - note that regions that
are naturally liable to change (e.g. trees moving) will
have a very high sigma, and will consequently be very
tolerant of any changes in the scene.

5) Generate a binary image of all regions that contain
pixels that are outside your confidence range (e.g.
Pixel_Difference > 3*PixelSigma;

6) Track your moving image using something like blob
centroid.

Regards

Dave Robinson

Thanks for the reply!

I have normalized the rgb like this:
Rn = R(x,y) / sum(R(x,y)) G(x,y)) B(x,y));

where Rn is the normalized value for Red, two similar
operation for Green and Blue.


I plan to just extract the first 10 frames and calculate
the mean pixel value from these frames, and create a new
image for which will be set as background. Is this good
enough? What do you use the std. deviation for in this case?

You say I should choose a confidence level, for example
three times larger than sigma. Is the sigma the same as
the absolute difference between to pixels?


Thanks, Helmer.

"Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
message <frb7ii$fpa$1@fred.mathworks.com>...
> Thanks for the reply!
>
> I have normalized the rgb like this:
> Rn = R(x,y) / sum(R(x,y)) G(x,y)) B(x,y));
>
> where Rn is the normalized value for Red, two similar
> operation for Green and Blue.
>
>
> I plan to just extract the first 10 frames and calculate
> the mean pixel value from these frames, and create a new
> image for which will be set as background. Is this good
> enough? What do you use the std. deviation for in this
case?
>
> You say I should choose a confidence level, for example
> three times larger than sigma. Is the sigma the same as
> the absolute difference between to pixels?
>
>
> Thanks, Helmer.

Your equation for normalizing RGB is usually pretty good, a
better one - but takes longer to compute is

REDNESS = Red/(sqrt(Red^2 + Green^2 + Blue^2))

Of course similar equations hold for GREENNESS and BLUENESS

Sigma(x,y) is the standard deviation derived from your
empty background image phase.

10 frames seems enough to start the process with - however
I would be tempted to continue the accumulation of
background statistics using any frame that didn't contain a
moving object - just a personal quirk perhaps. Obviously
the more frames you accumulate the better the statistics.

During operation time you pull in a frame. If you subtract
this frame from your mean background image, you can then
test if the pixel difference is statistically meaningful -
or whether it can be explained by the ambient noise level
at that pixel location. If, for example, the difference
value is smaller than say 3 times the standard deviation it
is unsafe to conclude that the difference is significant,
and you should declare this a null difference. If on the
other hand the difference is greater than the 3 sigma
threshold, then the difference can be safely declared as a
real effect, and treated as a change in the image. This
technique has the advantage that in regions where the
background colour does vary considerably between blank
frames, then the standard deviation is a lot higher, so to
be identified as a real change, the variation must be that
much greater than in a relatively stable zone.

From memory, the equations for continuously updating the
estimate for the mean and standard deviation of an
indefinite sequence of numbers has been outlined on this
forum several times.

Hope that helps

Dave Robinson

I might be asking a lot of silly questions now :)

I just want to make sure I got everything right.

Computing the mean pixel intesity is okay, and thereby
create a new image set as the background.

The sigma on the other hand, I know how to calculate it for
a set of values, but how to do it for a set of images? Do I
calculate the std.dev. for a set, in this case with the
size of 10 values, of pixels containing of all pixels from
position (1,1), and then do the same with a set of pixels
from position (1,2) until position (n,m). And then do a
std.dev. on all of these values again. So in the end I have
just one value containing the std.dev, or can I do a mean
of all the std.dev values I get from each pixel-position.
Hope you understand what I meant here.

"Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
message <frbbrd$k4h$1@fred.mathworks.com>...
> I might be asking a lot of silly questions now :)
>
> I just want to make sure I got everything right.
>
> Computing the mean pixel intesity is okay, and thereby
> create a new image set as the background.
>
> The sigma on the other hand, I know how to calculate it
for
> a set of values, but how to do it for a set of images? Do
I
> calculate the std.dev. for a set, in this case with the
> size of 10 values, of pixels containing of all pixels
from
> position (1,1), and then do the same with a set of pixels
> from position (1,2) until position (n,m). And then do a
> std.dev. on all of these values again. So in the end I
have
> just one value containing the std.dev, or can I do a mean
> of all the std.dev values I get from each pixel-position.
> Hope you understand what I meant here.
>

In this application the following attempts to explain what
I mean by the mean and standard deviation images.

If your image frame is say n x m pixels, then you have n x
m mean values and n x m standard deviation values.

You need to visualize a sequence of say 10 frames as an n x
m parallel stream of 10 numbers, So you end up with a mean
image of n x m pixels and a standard deviation image of n x
m pixels, where the pixel of each of these statistical
images is generated from a sequence of 10 numbers.

Hope that helps

Regards

Dave Robinson

Thanks a lot!

You have been most helpful. Thanks for the quick replies.
Now it is back to the coding.

Helmer.

To Dave Robinson, and others.

I have some further question for the scholar :)

The result was actually not as I hoped it would be. The
values for the mean background and the std.deviation is
decimal numbers. The values of each frame I load and
eventually convert to binary is all integer numbers. Should
these be converted to decimals also? Should maybe each
frame processed be normalized as well?

What I get know is no motion at all, it does not registar
any movement. This maybe because the because the I have
decimals one place and integers another place, or that they
are not converted into the same datarange. So, that the
comparison of the values in a fram against the standard
deviation reveals too low values all over the image, which
indicates no movement.

I might be missing one step in my code here. If you want, I
can upload my m-files to a server, if you want to take a
look at the overall process.

Thanks, Helmer.

"Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
message <frdnhu$ies$1@fred.mathworks.com>...
> To Dave Robinson, and others.
>
> I have some further question for the scholar :)
>
> The result was actually not as I hoped it would be. The
> values for the mean background and the std.deviation is
> decimal numbers. The values of each frame I load and
> eventually convert to binary is all integer numbers.
Should
> these be converted to decimals also? Should maybe each
> frame processed be normalized as well?
>
> What I get know is no motion at all, it does not registar
> any movement. This maybe because the because the I have
> decimals one place and integers another place, or that
they
> are not converted into the same datarange. So, that the
> comparison of the values in a fram against the standard
> deviation reveals too low values all over the image,
which
> indicates no movement.
>
> I might be missing one step in my code here. If you want,
I
> can upload my m-files to a server, if you want to take a
> look at the overall process.
>
> Thanks, Helmer.

Unless there is a very good reason for remaining in UINT 8
format, I normally switch to doubles as soon as any image
enters the computer (see my response to your other thread),
and only convert back when/if I need to display the result.

Puzzled with your comment about normalizing your input
frames. If you mean should you convert it to Normalized RGB
then yes ALL calculations should be done in the Normalized
colour domain. (these need to be doubles anyway to do the
division in the colour normalization process) If you mean
rescale your dynamic range somehow, the im2double() - or
whatever it is called automatically maps your input 0 -->
255 range into a 0 --> 1 scale, which will mean everything
is correctly scaled for you.

Alas I don't have the image processing toolbox here, so
there is very little point in uploading the code - unless
one of the real guru's come on line.

Hope that helps

Dave Robinson

Hi.
I'm facing the same problem with the background subtraction.
I have a stable camera recording at 60fps an animal walking in the sagittal
plane in a indoor area. I'm lighting the area using a few tungsten lamp (working
at 60Hz). Despite that, analyzing the frames one by one I can still notice very
small changes of the bright (flickering). Is that normal?

Beside that, I used an algorithm for the background subtraction found at http://staff.science.uva.nl/~zivkovic/DOWNLOAD.html
It's working quite well but I still have problems with the extraction of a leg of
the animal (that has, more or less, the same color of the background). Anyone
can suggest me what I can do to improve the extraction of the animal?

By the way, for every video I have to set different parameters to make the
algorithm working well. But I would like to avoid that in order to use it in semi-
real time application.

> 1) Use a colour camera
>
> 2) Convert to Normalized RGB to remove the effect of
> illumination variation.
>
> 3) Over a protracted period, with no moving target in the scene, calculate the mean background intensity of each individual pixel, together with its associated standard
deviation.
>
> 4) Select a confidence level (e.g. 3xsigma) whereby the

> 5) Generate a binary image of all regions that contain
> pixels that are outside your confidence range (e.g.
> Pixel_Difference > 3*PixelSigma;
>
> 6) Track your moving image using something like blob centroid.
I have a question. I have tried to follow your step but I got some problems with defining pixel that are outside of confidence level.
I will post some code here to explain my ideas:
movie = aviread('fire_sample.avi',[1:300]);
% reading some avifile file that was acquired by web-cam
% first 10 frames with no object to obtain background statistics
f1=movie(1).cdata;
........
.........
f10=movie(10).cdata;
f_size = size(f1);
width = f_size(2);
height = f_size(1);
% estimating mean value intensity for background
sumOfFramesñ+f2+f3+f4+f5+f6+f7+f8+f9+f10; % for estimating background statistics mean
back_mean=sumOfFrames./10; % array for mean values based on 10 frame statistics
back_subtraction=(sumOfFrames-back_mean); %x-x
back_sd=((back_subtraction.^2)./10); %(x-x)^2/10 estimating standard deviation
D = 3;
 thresh=D*back_sd; thresh=double(thresh);
for n=1:length(movie);
    fr = movie(n).cdata;
    fr=double(fr);
red_diff = abs(double(fr(:,:,1)) - double(back_mean(:,:,1))); %defining absolute different in a red channal
green_diff «s(double(fr(:,:,2)) - double(back_mean(:,:,2))); %in a green
blue_diff= abs(double(fr(:,:,3)) - double(back_mean(:,:,3))); %in blue
%here I try to define pixels that are outside of %range
for i=1:width
    for j=1:height
        if (red_diff>thresh(:,:,1))
            M(i,j,1)=1;
        else
            M(i,j,1)=0;
        end
    end
end
% I also do it for green and blue channel then I %want to make smth like this MÊt(3,red %channel, green,blue);
for i=1:width
   for j=1:height
       if M>2
            CM=1;
        else
            CM=0;
       end
    end
end

% I think I will obtain a matrix M where pixels that are greater then thresh, equal to 1s and where it is less then thresh pixels value would be zero, then I should obtain binary mask where threshold would be done as follow if there is a change it least in two color channels we define that it is likely pixel was really changed
I have a lack of programming so I don't understand why every time when I try to define pixels that are outside of range, M(:,:,1) =0 (all matrix consists of zeros values). Could you please explain my mistakes? Also avi file was normalized before I run foreground detection. I would be appreciate for any help

"Alla Kim" <vissa22@mail.ru> wrote in message <guuclt$bg$1@fred.mathworks.com>...
> > 1) Use a colour camera
> >
> > 2) Convert to Normalized RGB to remove the effect of
> > illumination variation.
> >
> > 3) Over a protracted period, with no moving target in the scene, calculate the mean background intensity of each individual pixel, together with its associated standard
> deviation.
> >
> > 4) Select a confidence level (e.g. 3xsigma) whereby the
>
> > 5) Generate a binary image of all regions that contain
> > pixels that are outside your confidence range (e.g.
> > Pixel_Difference > 3*PixelSigma;
> >
> > 6) Track your moving image using something like blob centroid.
> I have a question. I have tried to follow your step but I got some problems with defining pixel that are outside of confidence level.
> I will post some code here to explain my ideas:
> movie = aviread('fire_sample.avi',[1:300]);
> % reading some avifile file that was acquired by web-cam
> % first 10 frames with no object to obtain background statistics
> f1=movie(1).cdata;
> ........
> .........
> f10=movie(10).cdata;
> f_size = size(f1);
> width = f_size(2);
> height = f_size(1);
> % estimating mean value intensity for background
> sumOfFrames?+f2+f3+f4+f5+f6+f7+f8+f9+f10; % for estimating background statistics mean
> back_mean=sumOfFrames./10; % array for mean values based on 10 frame statistics
> back_subtraction=(sumOfFrames-back_mean); %x-x
> back_sd=((back_subtraction.^2)./10); %(x-x)^2/10 estimating standard deviation
> D = 3;
> thresh=D*back_sd; thresh=double(thresh);
> for n=1:length(movie);
> fr = movie(n).cdata;
> fr=double(fr);
> red_diff = abs(double(fr(:,:,1)) - double(back_mean(:,:,1))); %defining absolute different in a red channal
> green_diff ?s(double(fr(:,:,2)) - double(back_mean(:,:,2))); %in a green
> blue_diff= abs(double(fr(:,:,3)) - double(back_mean(:,:,3))); %in blue
> %here I try to define pixels that are outside of %range
> for i=1:width
> for j=1:height
> if (red_diff>thresh(:,:,1))
> M(i,j,1)=1;
> else
> M(i,j,1)=0;
> end
> end
> end
> % I also do it for green and blue channel then I %want to make smth like this M?t(3,red %channel, green,blue);
> for i=1:width
> for j=1:height
> if M>2
> CM=1;
> else
> CM=0;
> end
> end
> end
>
> % I think I will obtain a matrix M where pixels that are greater then thresh, equal to 1s and where it is less then thresh pixels value would be zero, then I should obtain binary mask where threshold would be done as follow if there is a change it least in two color channels we define that it is likely pixel was really changed
> I have a lack of programming so I don't understand why every time when I try to define pixels that are outside of range, M(:,:,1) =0 (all matrix consists of zeros values). Could you please explain my mistakes? Also avi file was normalized before I run foreground detection. I would be appreciate for any help

Please help me. I have already changed loop statement as follows:
             
               R(red_diff>78)=1;
               R(red_diff<=78)=0;
               G(green_diff>78)=1;
               G(green_diff<=78)=0;
               B(blue_diff>78)=1;
               B(blue_diff<=78)=0;
               

RGB=R+G+B;
CM=zeros(height,width);

CM(RGB>=2)=1;
CM(RGB<2)=0;
But may be my threshold is too low, the result is not good and can really detect any motion.

"Dave Robinson" <dave.robinson@somewhere.biz> wrote in message <fr8rvq$rfs$1@fred.mathworks.com>...
> "Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
> message <fr8o5t$m76$1@fred.mathworks.com>...
> > I am conducting a project which consists of object
> > tracking. There I want to have a good segmentation of the
> > moving objects from the background. The background may
> have
> > noise-factors like light changes, and wind in trees.
> >
> > What kind of segmentation is best? Should I use just
> > background subtraction, or is it best with motion
> > estimation?
> >
> > Doeas anybody have some thoughts on this subject?
> >
> > Thanks, Helmer.
>
> 1) Use a colour camera
>
> 2) Convert to Normalized RGB to remove the effect of
> illumination variation.
>
> 3) Over a protracted period, with no moving target in the
> scene, calculate the mean background intensity of each
> individual pixel, together with its associated standard
> deviation.
>
> 4) Select a confidence level (e.g. 3xsigma) whereby the
> difference between the mean image and the current image is
> deemed to be statistically similar - note that regions that
> are naturally liable to change (e.g. trees moving) will
> have a very high sigma, and will consequently be very
> tolerant of any changes in the scene.
>
> 5) Generate a binary image of all regions that contain
> pixels that are outside your confidence range (e.g.
> Pixel_Difference > 3*PixelSigma;
>
> 6) Track your moving image using something like blob
> centroid.
>
> Regards
>

> Dave Robinson

Hello Dave,

I tried the same exact method you've put down here, but the normalized RGB image I'm getting is devoid of any worthwhile information of the image.

Have a look here :

http://img524.imageshack.us/img524/7079/sdafdfs.png

This is not how it should look, right?

"Abhishek " <highlytoxicboy@gmail.com> wrote in message <h00vb2$p8i$1@fred.mathworks.com>...
> "Dave Robinson" <dave.robinson@somewhere.biz> wrote in message <fr8rvq$rfs$1@fred.mathworks.com>...
> > "Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
> > message <fr8o5t$m76$1@fred.mathworks.com>...
> > > I am conducting a project which consists of object
> > > tracking. There I want to have a good segmentation of the
> > > moving objects from the background. The background may
> > have
> > > noise-factors like light changes, and wind in trees.
> > >
> > > What kind of segmentation is best? Should I use just
> > > background subtraction, or is it best with motion
> > > estimation?
> > >
> > > Doeas anybody have some thoughts on this subject?
> > >
> > > Thanks, Helmer.
> >
> > 1) Use a colour camera
> >
> > 2) Convert to Normalized RGB to remove the effect of
> > illumination variation.
> >
> > 3) Over a protracted period, with no moving target in the
> > scene, calculate the mean background intensity of each
> > individual pixel, together with its associated standard
> > deviation.
> >
> > 4) Select a confidence level (e.g. 3xsigma) whereby the
> > difference between the mean image and the current image is
> > deemed to be statistically similar - note that regions that
> > are naturally liable to change (e.g. trees moving) will
> > have a very high sigma, and will consequently be very
> > tolerant of any changes in the scene.
> >
> > 5) Generate a binary image of all regions that contain
> > pixels that are outside your confidence range (e.g.
> > Pixel_Difference > 3*PixelSigma;
> >
> > 6) Track your moving image using something like blob
> > centroid.
> >
> > Regards
> >
>
> > Dave Robinson
>
> Hello Dave,
>
> I tried the same exact method you've put down here, but the normalized RGB image I'm getting is devoid of any worthwhile information of the image.
>
> Have a look here :
>
> http://img524.imageshack.us/img524/7079/sdafdfs.png
>
> This is not how it should look, right?

Without looking at your original image, I couldn't really make a judgement, put an full RGB image on the same site, and I might get a chance to look at it.

Dave Robinson

"Dave Robinson" <dave.robinson@somewhere.biz> wrote in message <h010gj$ch1$1@fred.mathworks.com>...
> "Abhishek " <highlytoxicboy@gmail.com> wrote in message <h00vb2$p8i$1@fred.mathworks.com>...
> > "Dave Robinson" <dave.robinson@somewhere.biz> wrote in message <fr8rvq$rfs$1@fred.mathworks.com>...
> > > "Torvald Helmer" <torvald.helmer@mathworks.com> wrote in
> > > message <fr8o5t$m76$1@fred.mathworks.com>...
> > > > I am conducting a project which consists of object
> > > > tracking. There I want to have a good segmentation of the
> > > > moving objects from the background. The background may
> > > have
> > > > noise-factors like light changes, and wind in trees.
> > > >
> > > > What kind of segmentation is best? Should I use just
> > > > background subtraction, or is it best with motion
> > > > estimation?
> > > >
> > > > Doeas anybody have some thoughts on this subject?
> > > >
> > > > Thanks, Helmer.
> > >
> > > 1) Use a colour camera
> > >
> > > 2) Convert to Normalized RGB to remove the effect of
> > > illumination variation.
> > >
> > > 3) Over a protracted period, with no moving target in the
> > > scene, calculate the mean background intensity of each
> > > individual pixel, together with its associated standard
> > > deviation.
> > >
> > > 4) Select a confidence level (e.g. 3xsigma) whereby the
> > > difference between the mean image and the current image is
> > > deemed to be statistically similar - note that regions that
> > > are naturally liable to change (e.g. trees moving) will
> > > have a very high sigma, and will consequently be very
> > > tolerant of any changes in the scene.
> > >
> > > 5) Generate a binary image of all regions that contain
> > > pixels that are outside your confidence range (e.g.
> > > Pixel_Difference > 3*PixelSigma;
> > >
> > > 6) Track your moving image using something like blob
> > > centroid.
> > >
> > > Regards
> > >
> >
> > > Dave Robinson
> >
> > Hello Dave,
> >
> > I tried the same exact method you've put down here, but the normalized RGB image I'm getting is devoid of any worthwhile information of the image.
> >
> > Have a look here :
> >
> > http://img524.imageshack.us/img524/7079/sdafdfs.png
> >
> > This is not how it should look, right?
>
> Without looking at your original image, I couldn't really make a judgement, put an full RGB image on the same site, and I might get a chance to look at it.
>
> Dave Robinson

Sorry, I should have uploaded the original image first. Anyways, here it is now :

http://img193.imageshack.us/img193/7710/picture8v.jpg

Here is the Normalized RGB form of the same image -

http://img193.imageshack.us/img193/5684/11210646.png

I'm using the following relation for conversion :

R=R/(R+G+B), etc.

Thanks for your help.
Cheers