I have a similar problem too. I use fmincon for a large problem (nearly half a million variables), with the following options/settings:
After every iteration the parallel pool goes idle for some time, and this idling time between the iterations is gradually increasing with iterations. I do not know why and how to deal with this.
I tried it on different versions of MATLAB (R2016b, R2018b, R2020a and R2020b), but the same problem persists.
Any answers as of why this is happening and how to fix this ?