The new set of faces refer to indices of vertices actually used by the faces. So it is necessary to use the following syntax:
[V2 F2] = mergeCoplanarFaces(vertices, K);
The demo file is somewhat outdated, I will update it, and fix the doc of the mergeCoplanarFaces function.
Concerning the tolerance value, it is used for comparing normalised normal vectors of planes containing faces. There can be some numerical issues, but using 1e-4 should be fine.