Parallel computation has overhead to transfer data to the workers and to transfer results back. Because of that overhead, it is most effective when the amount of computation is relatively large compared to the amount of data transferred. If you have a short computation then the overhead can swamp any increase in performance.
If the objective function does heavy computation on large enough matrices, MATLAB will automatically call out to high performance multi-threaded libraries. In versions up to R2016a, each parallel worker can be allocated only a single core, so within each parallel worker, the calls to the high performance libraries could use only a single core, which can degrade performance. Four mldivide \ operations in parallel would, for example, typically be worse performance than doing four sequential \ operations because each of the parallel \ operations would have been allocated only one core.
Starting in R2016b it became possible to configure the parallel pools to allocate more than one core per worker. This reduces the number of parallel tasks, but can improve performance for each task. This does not happen by default, though.
Computation in parallel is not always faster than serial computation; sometimes it is only the same, and sometimes the overhead makes it much worse than serial.