"Passing by reference vs value"
Neither. MATLAB is more intelligent that either of those. MATLAB uses something called "copy on write", which essentially means that is passes by reference unitl the data is changed, at which point it makes a copy. This is explained in the blogs: "MATLAB makes copies of arrays it passes only when the data referred to changes"
What does this mean for your code? In 99.99% of cases you should just use the simplest syntax, i.e. simply pass the variables and let MATLAB's memory management take care of how the variables are stored. In your example just pass a and b, no copies will be made.
In any case, TMW discourages writing code to take advantage of specific JIT features. The philosophy is that the JIT engine is written to optimize your code, not the other way around.