I understand that you want to know why the functions mattest and ttest2 behave differently for the same input.
Both functions accept the two primary inputs as either vectors or matrices. In your case, you indicated that you wanted to work on matrices, in which case, they are required to have the same number of columns. There are a couple of differences in the standard syntax for mattest and ttest2 functions that contribute to this difference in behavior for the same inputs.
Firstly, mattest performs the test assuming the two samples have unknown and unequal variances unless the VarTypeValue argument is specifically set to 'equal'. On the other hand, ttest2 conducts a test using the assumption that the two samples are from normal distributions with unknown but equal variances.
Secondly, unless the 'Dim' Name-Value pair in ttest2 is explicitly set to 2 (to test the row means), it assumes it to be the first nonsingleton dimension.
Here is an example using both functions to generate the same p-values:
x = rand(2,5);
y = rand(2,5);
p1 = mattest(x,y)
[~,p2,~,~] = ttest2(x,y,'vartype','unequal','dim',2)