I found a way to reproduce the other part of the problem. Replace the DoStuff function in the above code by a real calculation instead of a pause and now it queues instead of cancelling, both if you click on the other radiobutton or the pushbutton while the calculation is going on. Unfortunately, all this points to Matlab not actually being able to do what I wanted by using these properties (and at least this second problem looks like a bug to me). [Edit: actually, adding a "pause(0.1);" after the calculation seems to "fix" this part of the problem, and see the comment above for a somewhat obvious workaround to the other part.] Any comments?
% Set mouse pointer as busy.
set(handles.figure1, 'Pointer', 'watch');
drawnow; % so the mouse pointer actually changes...
% Display something.
x = inv(rand(4000));
set(handles.text3, 'String', 'Done!');
% Reset mouse pointer.
set(handles.figure1, 'Pointer', 'arrow');