I found the root cause of this via an strace of the MATLAB process. It revealed this tidbit shortly after the attempt to run the first system() invocation:
553435 ioctl(3, SNDRV_TIMER_IOCTL_SELECT or TIOCSPGRP, ) = -1 ENOTTY (Inappropriate ioctl for device)
The "ENOTTY" means that the ioctl() call is inapplicable because the session is not associated with a pseudo-tty. And the reason that a poll() error comes back is because the thread reporting the error is not the one actually doing the fork/exec - it just sent a message to the thread that does. And so when the fork/exec thread fails, the poll() waiting for the response from that thread is interrupted.
The fix is to use the "-t" option for SSH, that is:
ssh -t -X user@machine 'matlab -nodesktop'
This forces the SSH server to allocate a pseudo-tty to the MATLAB process' environment whether it thinks it needs it or not, and so the ENOTTY error is avoided, and so allows the first system() call, and all subsequent system() calls, to work as expected.