How can I extract foreground object(human) with depth information from a RGB-D video sequence?

I have a RGB-D camera video sequence as input, I have used MoG foreground detector on RGB information and I am able to extract foreground object from each frame. But I need the depth information of the pixels as well.

When depth information is fed as input to MoG detector, and various parameters for MoG, I am unable to fetch the foreground object.

Could you please tell me how to use depth image to extract fg from bg?

Any segmentation technique can be used.