McNemarextest

McNemar's Exact Test.

Description

This m-file performs the conditional as well as the Chi-squared corrected for discontinuity McNemar's exact test for two dependent (correlated) samples that can occur in matched-pair studies with a dichotomous (yes-no) response. Dependent samples can also occur when the same subject is measured at two different times. It tests the null hypothesis of marginal homogeneity. This implies that rows totals are equal to the corresponding column totals. So, since the a- and the d-value on both sides of the equations cancel, then b = c. This is the basis of the proposed test by McNemar (1947). The McNemar test tests the null hypothesis that conditional on b + c, b has a binomial (b + c, 1/2) distribution.
 
The proper way to test the null hypothesis is to apply the one-sample binomial test. If there is no association between b and c values, then the probability is 0.5 that the sample 1 and sample 2 pair falls in the
upper-right cell and 0.5 that it falls in the lower-left cell, given that the pair falls off the main diagonal.

File needs to input data vector defined by the observed frequency cells [a,b,c,d], desired test [t = 1, one-tail; t = 2, two-tail (default)], and significance level (default = 0.05).

It outputs a table with the proportion of success for the dependent samples and the P-value.

• MATLAB release: MATLAB 7 (R14)

[david vexler]

Actually, it is conditional exactly mcnemar test. It is far from inefficient programm.
line 137 to 150 can be change to

P = 1;
nf = prod(1:m);
for i = 0:x
    fac1 = prod(1:i);
    fac2 = prod(1:m-i);
    P=P-(p.^i)*nf*(1-p).^(m-i)./(fac1*fac2);
end;

[J. Z.]

Dear Mr. Vexler. I'm sorry to tell you that your very rigorous qualification in nothing contradicts the results of this m-function. You don't give or statistically support the fundamentals. Your observation only helps to reduces the program typing. Really, this m-function has excellently helped in many applied problems.

[I Y]

There are two things this programm ingnored.
First, there are two kind of 'one tail test',either sample 2 greater than 1 or sampel 1 great than 2.
second, the P-value should be constrained to less than 1.

david's code is also king long and inefficient. Following works well in R13

x=max(b,c);
P=0;
for i =x:m
    P=P+(p.^i)*(1-p).^(m-i)*nchoosek(m,i);
end;

[albert Miller]

Agree with david. He is right, the function only performs conditional exact test.

[HR Kang]

It will be useful if output argument feature is added, like following:
function res = McNemarextest(v, t, alpha)
...
if t==1
 if P>alpha
  res = 0;
 else
  if b>d, res=1; else res=-1; end
 end
else
 if P>alpha
  res = 0;
 else
  res = 1;
 end
end

In this way, res will be:
in 2-tailed test, 0 if not significant, 1 if significant
in 1-tailed test, 0 if not significant, 1 if b>d significantly, -1 if b<d significantly.

[HR Kang]

There was a typo in the above comment:
the second 'if P>alpha' should be replaced with 'if P>alpha/2'

[HR Kang]

Another mistake: 'b>d' --> 'c>b'
Then res==1 will mean sample 1 having more success, and res==-1 the opposite.

[Skynet]

 

[Tak-Shing]

By convention, results are deemed significant if P < alpha. McNemarextest uses P <= alpha which differs from common usage.

Secondly, for the two-tailed test, if b = c then p(b) = p(c) will be summed twice, yielding an invalid P.

Thirdly, the use of "ELSE expression" is possibly inappropriate (suggested fix: "ELSEIF expression" or just a plain "ELSE", depending on context).

Finally, IY's code can be further shortened:
P = 1-binocdf(max(b,c)-1,m,p);

[Julio]

First of all, I would like to thank the authors for sharing this file. I am using the McNemar test to compare two classifiers, as suggested in (Kuncheva, Combining Patten Classifiers, 2004; and Guyon et al, Feature Extraction - Foundations and Applications, 2006). However I would like to raise a few points:
1) In a one-tailed test, the p-value should be halved, not the significance level against which the p-value is compared. Check the source code for the ttest() function and you will see an example of correct implementation of one-tailed test;
2) A one-tailed test should give a different p-value if the input is swapped, which in this case means swapping the second and third element of the "v" input. The implementation of the one-tailed version is INCORRECT (I am still looking for a way to implement a one-tailed McNemar test)
3) The second part of the program is displaying the "P" variable (calculated in the first part) when it should be displaying the "p" variable

I hope this helps.