Ok fair enough. But the other issue I have is that the rect used to specify the position is not consistent with java coordinates (mentioned in your code). These have coordinates at top left - yours is bottom left and for some inexplicable reason (0,0) is off screen in your coordinate system. I see there are "estimated uncertainties" but where do these come from? Should be just a simple translation/reflection between the two with zero uncertainty. I can reverse engineer the offset and subtract it out but will this be different each time?
Murray - that's the entire point: the only way to interactively select a screen-capture box anywhere on the desktop (not just within a Matlab figure) is to create an almost-transparent figure that spans the entire desktop area, and then let the user click-and-drag within it. You don't need to find the edges at all - simply drag a box of your requested screen-capture area and then the almost-transparent window will disappear and you will get your desired screen-capture.
A technical explanation is provided here: http://undocumentedmatlab.com/blog/screencapture-utility
The mouse click window for interactively selecting a target area doesn't appear to work very well. A window appears and then disappears (almost) when the prompt comes up but it remains almost invisible - pretty much impossible to find the edges.