When figures are docked, the position always start in the origin (i.e. 1,1), you could detect this location by using gcf like this:
1 1 623 237
In contrast, when you open a figure, it will not be in the origin:
560 528 560 420
Thus, if you loop through your figures and check the origin, you can select those which have 1,1 and close those ones, and leave the other ones open (or vice versa).