I get a warning not an error if I try to save a workspace with an non-serialisable java variable:
>> socket = java.net.DatagramSocket;
Warning: java.net.DatagramSocket@54d46c8 is not serializable
Warning: Variable 'socket' was not saved. Variables of type 'java.net.DatagramSocket' (or one of its members) are not supported
for this MAT-file version. Use a newer MAT-file version.
The warning is a bit misleading, since it appears using the latest Mat-file version (7.3).
Anyway, if you want to exclude all the java variables from save, you could do this:
vars = whos;
varsnojava = vars(arrayun(@(var) ~isjava(evalin('base', var.name)), vars))
Even better would be to have all variables you want to save as fields of a single structure and simply save that structure (with the -struct option of save). Ultimately, this would be more robust as you wouldn't have to worry about any temporary, or non-serialisable, or any other undesired variable in your workspace when you do the save.