I'd first define a mean green color. Then convert to lab color space and calculate a delta E image. Then threshold the delta E to find out where the delta E is more than some certain amount. Those will be the non-green pixels. See my delta E demo athttp://www.mathworks.com/matlabcentral/fileexchange/?term=authorid%3A31862 and adapt it. I do this kind of thing all the time and yours is not that hard. Note, you cannot use automatic thresholding despite what some novices at image processing may suggest because you need to findall amounts of brown from 0% to 100%.