The Computer Vision System Toolkit offers tools for processing video in ways that tend to avoid filling up memory.
Beyond that, we would need to look at your code.
One thing to avoid: do not read the entire video in at the same time if you can reasonably avoid doing so. Read one frame at a time, when you need the frame, keeping only a limited number of previous frames in memory.
Another thing: if you are displaying plots as you go, make sure you delete the old graphics objects instead of just allowing them to accumulate in memory. Use calls such as cla() to clear the axes; use delete() on plot objects when you finish with them. Avoid plotting one point at a time; if you are drawing a lot of disconnected points, gather them in a scatter() object and update the 'XData' and 'YData' properties to add more points or move the existing points.
