For anyone interested, I ended up solving this myself with the help of a coworker. Just cut results at 0 to only show negatives:
[c, lags] = xcorr(zscore(v1), zscore(v2), 25, 'coeff');
% cut results by half
lags = lags(lags<0);
c = c(lags<0);
% Get the index of the max value and use it to find the (now more precise) lag
[~,index] = max(c);
t = lags(index);