File Exchange

image thumbnail

Fleiss

version 2.0.0.0 (15 KB) by Giuseppe Cardillo
Compute the Fleiss'es kappa for multiple raters

16 Downloads

Updated 17 Mar 2018

From GitHub

View Version History

View license on GitHub

Fleiss'es kappa is a generalisation of Scott's pi statistic, a
statistical measure of inter-rater reliability. It is also related to
Cohen's kappa statistic. Whereas Scott's pi and Cohen's kappa work for
only two raters, Fleiss'es kappa works for any number of raters giving
categorical ratings (see nominal data), to a fixed number of items. It
can be interpreted as expressing the extent to which the observed amount
of agreement among raters exceeds what would be expected if all raters
made their ratings completely randomly. Agreement can be thought of as
follows, if a fixed number of people assign numerical ratings to a number
of items then the kappa will give a measure for how consistent the
ratings are. The scoring range is between 0 and 1.
Syntax: fleiss(X,alpha)

Inputs:
X - square data matrix
ALPHA - significance level (default = 0.05)
Outputs:
- kappa value for the j-th category (kj)
- kj standard error
- z of kj
- p-value
- Fleiss'es kappa
- kappa standard error
- kappa confidence interval
- k benchmarks by Landis and Koch
- z test

Example:
An example of the use of Fleiss'es kappa may be the following: Consider
fourteen psychiatrists are asked to look at ten patients. Each
psychiatrist gives one of possibly five diagnoses to each patient. The
Fleiss' kappa can be computed from this matrix to show
the degree of agreement between the psychiatrists above the level of
agreement expected by chance.
x =
0 0 0 0 14
0 2 6 4 2
0 0 3 5 6
0 3 9 2 0
2 2 8 1 1
7 7 0 0 0
3 2 6 3 0
2 5 3 2 2
6 5 2 1 0
0 2 2 3 7

So there are 10 rows (1 for each patient) and 5 columns (1 for each
diagnosis). Each cell represents the number of raters who
assigned the i-th subject to the j-th category
x=[0 0 0 0 14; 0 2 6 4 2; 0 0 3 5 6; 0 3 9 2 0; 2 2 8 1 1 ; 7 7 0 0 0;...
3 2 6 3 0; 2 5 3 2 2; 6 5 2 1 0; 0 2 2 3 7];

Calling on Matlab the function: fleiss(x);

Answer is:

kj: 0.2013 0.0797 0.1716 0.0304 0.5077

s.e.: 0.0331

z: 6.0719 2.4034 5.1764 0.9165 15.3141

p: 0.0000 0.0162 0.0000 0.3594 0

------------------------------------------------------------
Fleiss'es (overall) kappa = 0.2099
kappa error = 0.0170
kappa C.I. (95%) = 0.1767 0.2432
Fair agreement
z = 12.3743 p = 0.0000
Reject null hypotesis: observed agreement is not accidental

Created by Giuseppe Cardillo
giuseppe.cardillo-edta@poste.it

To cite this file, this would be an appropriate format:
Cardillo G. (2007) Fleiss'es kappa: compute the Fleiss'es kappa for multiple raters.
http://www.mathworks.com/matlabcentral/fileexchange/15426

Cite As

Giuseppe Cardillo (2021). Fleiss (https://github.com/dnafinder/Fleiss), GitHub. Retrieved .

Comments and Ratings (12)

Jenil Shah

Ana

Giuseppe Cardillo

No No this file is an open code: you can modify it as you want providing my acknowledgment

Daniel Golden

Thanks for contributing this, Giuseppe. I moved the submission to my own git repository in order to make a few changes and facilitate making changes in the future: https://github.com/drdan14/matlab_fleiss_kappa

Let me know if that's OK, or whether you'd prefer to use your own git repository that others can fork.

My modifications are in the current master and your FEX version from 23 Dec 2009 is https://github.com/drdan14/matlab_fleiss_kappa/tree/v2009.12.23

Giuseppe Cardillo

no you are right

nicolas

Sorry, my mistake: pj are effectively different. But kj and zj are not.

With j=2, sum(x.*(m-x)) yields two identical values. As observers can choose only between category 1 or category 2, n votes for cat 1 induce m-n votes for cat 2.

Parameter b=pj.*(1-pj) yields also 2 identical values with j=2.

I am wrong ?

Giuseppe Cardillo

Of course, no. Pj is function of Kj and K1=K2 if and only if sum(x.*(m-x)) are equal (m is the numbers of raters).

nicolas

Hi,
I tried your code with only 2 categories (so j=2). For all various set of data tested, I always got identical values for k1 and k2, p1 and p2...

Is it theoretically normal or not ? when j=2, k1=k2 ?

Thank you for your answer.

Adrian ADEWUNMI

With much pleasure, I am pleased to announce I have solved the problem .

Thanks to Giuseppe Cardillo for this matlab function.....good job.

Adrian ADEWUNMI

Whenever I imput any other matrix than a 5 x 10 matrix into matlab, using your function "fleiss(X)"it gives an error message as follows:

EDU>> fleiss(X)
??? Error using ==> fleiss at 107
The raters are not the same for each rows

Can you tell me how to fix this?
Thx

Giuseppe Cardillo

The Fleiss'es kappa is an overall valuation of agreement. It doesn't recognize differences among raters. I think that this can be done using Cohen's kappa.
An example of the use of Fleiss'es kappa may be the following: Consider 14 psychiatrists are asked to look at ten patients. Each psychiatrist gives one of possibly five diagnoses to each patient. The Fleiss'es kappa can be computed to show the degree of agreement among the psychiatrists above the level of agreement expected by chance.

Amy Graham

I think this m-file is to work with rates not raters.

MATLAB Release Compatibility
Created with R2014b
Compatible with any release
Platform Compatibility
Windows macOS Linux

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!