File Exchange

image thumbnail

hhentschke/measures-of-effect-size-toolbox

version 1.6.0.0 (1.36 MB) by Harald Hentschke
Computes diverse effect size statistics including confidence intervals

42 Downloads

Updated 26 Apr 2018

GitHub view license on GitHub

Editor's Note: This file was selected as MATLAB Central Pick of the Week

The Measures of Effect Size (MES) Toolbox is a set of functions which compute a wide range of effect size statistics. The four main toolbox functions cover common analysis designs, including two-sample-, oneway- and twoway- data sets as well as categorical data (tables). Data may be repeated-measures (within-subjects).
MES for contrasts can be computed. Confidence intervals are generated for the large majority of MES, either via bootstrapping or by analytical computation, in part via noncentral t, Chi square or F distributions.
Effect size statistics are complemented by t/Chi square/F statistics and/or full ANOVA tables, which are also provided as output variables.
The toolbox was developed by Harald Hentschke (University of Tübingen) and Maik Stüttgen (University of Bochum) and is accompanied by a paper (Hentschke and Stüttgen, Eur J Neurosci 34:1887-1894, 2011).
Among the ESM available are:
Hedges’ g
Glass’ delta
requivalent (point-biserial correlation)
common language effect size
Cohen’s U1
Cohen’s U3
receiver-operating characteristic
right/left tail ratio
rank-biserial correlation
standardized mean differences for contrasts
eta squared
partial eta squared
omega squared
partial omega squared
risk difference
risk ratio
odds ratio
phi
sensitivity
specificity
positive predictive value
negative predictive value
binomial effect size display
Cramer’s V

For more details please see the documentation.

Comments and Ratings (32)

@Ruggero I realized belatedly that Cliff's delta is in fact AUROC rescaled: Cliff's delta = 2 * AUROC - 1. As the conversion from one to the other is trivial, there is no point in implementing Cliff's delta alongside AUROC; instead, a note on the relation between the two has been added to the documentation.

Ruggero, Cliff's delta had not been implemented, but I have done so today, it's fairly simple. However, I'm on a kind of sabbatical which does not permit me to go through the machinations of updating the documentation soon, so if you'd like to have the undocumented version please drop me an email. Will officially include it with the next update (autumn, likely). Other than that, I can recommend you to use AUROC - it is also very robust and possibly better known than Cliff's delta.

Hi Harald,
I was wondering if in the package there is an implementation of Cliff's Delta, perhaps under a different name.
I saw you implemented Cohen's U1 and U3, which are indeed very similar non-parametric MES.
Are you planning to add Cliff's Delta?
Thanks in advance and congrats for the great package!

Bryson, it is necessary for data which are dependent (repeated-measures) with respect to any of the two factors. The code (in mes2way) relies on the data from any group forming a contiguous block; moreover "...it is assumed that input data X are sorted according to subjects; that is, within each group data for subjects are listed in the same order" (see explanation of optional input arg 'isDep'). So, the rationale is that if the data are not provided in groupwise contiguous blocks the user is probably not aware of the second of above-mentioned requirements, and must therefore order them before running mes2way.

I was getting the error below and came across this explanation. Is there any reason that I NEED to sort my data for this function other than "it appears possible that the data are messed up?" I can sort the data, but is it necessary for any other part of the function to run?

% if group is unsorted in the sense that samples from any group do not
% form a contiguous block it appears possible that the data are messed
% up, so better generate an error here and force the user to rethink (and
% sort) the data
isContiguousGroups=sum(any(diff(group),2))==nGroup-1;
if ~isContiguousGroups
error([mfilename ' expects samples from each group to form contiguous blocks. Please sort your data accordingly']);
end

Yukyung, the short answer is that you can't do it with mes.m alone. You'll have to write some wrapper code that e.g. loops through the column combinations of your matrices A and B and calls mes.m in each loop iteration. However, based on your example I wish to emphasize two points.
First, terminology. rank-biserial correlation is NOT a Pearson's type of correlation between variables A and B as in your example; rather, it is a correlation between a ranking and a dichotomy (e.g. group membership). Please see the introductory notes on p. 7 of the documentation (paragraph "The 'correlation' type of analyses"), in which we try to explain the difference with a concrete example.
Second, input arguments A and B may in fact be multi-column arrays, the simple reason being that in practice we often have more than one pair of groups that we want to compare, and that running multiple comparisons within mes.m is computationally much more efficient than having to call mes.m repetitively (the example in Fig. 4 in our 2011 paper is a case in point). For multiple comparisons to work, input variables A and B must have the same number of columns but may have a different number of rows (see the explanation of input arguments in mes.m). That is the inverse of what is required for input into e.g. corr, from which you seem to have drawn your example.

yukyung shin

Hello. I like your toolbox. I am doing rank-biserial correlation such as 'rbcorr'. If the size of the matrix A is nXa and the size of the matrix B is nXb, I want to have axb as a function of the corr in matlab. But your code is structured and I do not know how to do it. How can I do?

Hi everyone, I ported the code to GitHub; it will be automatically updated here from GitHub. If you want to report any issue or have ideas for enhancements, I'd appreciate you doing so on Github, which is ideally suited for discussion and collaboration.
Just a quick response to the latest issue: Rainer, thanks for spotting this. I see your point, but according to Nakagawa and Cuthill (2007), "...note that when a paired designed is used, n 1 = n 2 = n so that the denominator can be written as 8(n–1) – 1..." (p. 599), which is the current implementation. That's about the only reference to bias correction for Hedges' g in the case of dependent data I have found so far. Please let me know (see above) if I'm mistaken or there are other grounds for changing the code.

Hello, I really like your toolbox, would love to see it in R ;-)
Maybe I am mistaken, but it seems as if your bias correction in mes.m is not correct for dependent data. You comment:

% correct for bias due to small n (both dependent and independent
% data, Kline 2004 (p. 102, 106); Nakagawa & Cuthill 2007)
biasFac=(1-(3./(4*n1+4*n2-9)));
es=es.*biasFac;

But this is not the case, the general form is j = 1 - (3/(4 *df - 1)), where df are the degrees of freedom. In case of independent data it is df=(n1+n2-2) [then j=biasFac=(1-(3./(4*n1+4*n2-9)))], but for the dependent data case df=npairs-1, where npairs=n1=n2 (Borenstein, 2009; Kline, 2004, 2013; Rosenthal, 1994, and also Nakagawa & Cuthill, but they only explain it in the text).

It would appreaciate if you could check this.

Thanks for spotting the inconsistency - and no worries: version 1.4, described as the most recent version in code and documentation, is truly the most recent version. Version 1.5 must have resulted from me inadvertently uploading the same code twice on 5 Jan 2015, and there is no way for me to set the Update version on the File Exchange site one tick back (I think). I'll try to rectify this in the next update.

jkr

jkr (view profile)

A welcome toolbox that I look forward to exploring.

An administrative note: whether you choose to install the Toolbox directly (option under Download) or download the zip instead and expand and install it yourself, you get version 1.4. VersionHistory_MESToolbox.pdf covers only through v 1.4. Each code module references v 1.4. The comment below:
"1.5
- mixed within/between analyses are now possible with unequal sample sizes along the between-subjects factor
- minor edits of comments in code and documentation
See VersionHistory_MESToolbox.pdf for details"
Is all that is apparently available describing the difference. It is not clear to me how to get v.1.5.
I doubt this will be a problem for me - just wanted to point it out as I found it confusing.

Yamil Vidal

I got it...
You apply correction for bias due to small sample size.
Thanks a lot.

korkam

korkam (view profile)

Bob Spunt

mes1way is intended for statistics on two or more groups, similar to a oneway ANOVA. If you read the first 20 or so lines of the help you'll know how to arrange the input into mes1way (which is a trifle more complicated than with mes).

Oliver Kumar

Nice toolbox!
Is there a way to yous mes1way to compare two variables A and B. Like it is for mes? Thanks

This would indeed be a worthwile addition (thanks for the hint), but for a lack of time we have to restrict ourselves to maintenance and minor edits of the code, at least for now.

Julie

Julie (view profile)

Great toolbox - have you thought of adding one for heterogeneous variances (e.g. Johnston et al 2004 http://www.ncbi.nlm.nih.gov/pubmed/15058860)?

The example you refer to in the documentation has a line break in an unfortunate place, namely right after the second minus sign in the contrast weights matrix. I presume that upon pasting the example in the command window the code line read (incorrectly)

'cWeight',[1 -1 0; 1 0 - 1]

instead of

'cWeight',[1 -1 0; 1 0 -1]

(note the space after the second minus sign in the incorrect line), which is the likely cause of the error. We'll make sure that in the next version of the documentation code line breaks are more reasonably placed.

Jay Buckey

Ran this example in the manual:
mes1way(com_post,'partialeta2','group',group,'cWeight',[1 -1 0; 1 0 - 1],'isDep',1,'nBoot',10000)

and got this error:
Dimensions of matrices being concatenated are not consistent.

Dominika I

Nolan Conaway

korkam

korkam (view profile)

Hi Pat,

it seems that your data are not sorted in a way mes2way expects them to be, although your example suggests they should be. Anyways, let's assume

tmpData=
[1.4 1 1
1.5 1 2
2 1 3
...
is a N by 3 array containing the dependent variable (whatever it is that you record) in the first column and the numbers coding for the levels of the first and second factor in the 2nd and 3rd column, respectively (in your example, groups and blocks). The following lines

tmpData=sortrows(tmpData,[2 3]);
x=tmpData(:,1);
group=tmpData(:,[2 3]);

should do the trick.

Pat

Pat (view profile)

Hello Harald,

I'm trying this toolbox with a two-way ANOVA, with 3 groups and 40 blocks. I'm not sure how to code the X and g matrices. Right now I have something like:

data Groups Blocks # -->> not in the matrix

1.4 1 1
1.5 1 2
2 1 3
.
.
.
1.5 1 40
.
.
.
and so on for the other 2 groups

But it yielded this error:
??? Error using ==> mes2way at 335
mes2way expects samples from each group to form contiguous blocks. Please sort your data accordingly

Can you help me out,

Thanks

Pat

Zhiguo Ma

Excellent toolbox and document.
It makes effect size and its confidence iterval estimation easy to use and understand.
Thanks very much for your code.

Hello James, no, sorry, it's neither possible nor planned for the near future

James

James (view profile)

Is it possible to use this toolbox for ANOVAs with >2 factors?

Hi Jonathan, yes it does: Hedges' g is one manifestation of Cohen's D, maybe the most popular one. For more details on this, please see the documentation.

Jonathan

does this have Cohen's D, by some other name?? could it be added? at least in Psychology, its a widely used effect size estimate. Thanks!

Wow!
Very impressed with the well written code and manual. Thanks a bunch!

Updates

1.6.0.0

Ported code to GitHub (to allow for better collaboration on the code) and set up link from GitHub. Changed variable name 'table' in mes1way and mes2way to 'summaryTable'.

1.5.0.0

- mixed within/between analyses are now possible with unequal sample sizes along the between-subjects factor
- minor edits of comments in code and documentation

See VersionHistory_MESToolbox.pdf for details

1.4.0.0

- mixed within/between analyses are now possible with unequal sample sizes along the between-subjects factor
- minor edits of comments in code and documentation
See VersionHistory_MESToolbox.pdf for details

1.3.0.0

- implementation of exact confidence intervals for g1
- option to plot data in twoway analysis
- enhancement & correction of documentation

See VersionHistory_MESToolbox.pdf for details.

1.2.0.0

- added unstandardized MES (mean difference & contrasts)
- enhanced documentation accordingly
- bug fix in mes1way
- streamlined computation of t statistics in mes1way

The changes are explained in more detail in VersionHistory_MESToolbox.pdf

1.1.0.0

- some bug fixes
- additions to the documentation
- added example data (which were missing in the previous version, sorry)

Details are listed in VersionHistory.pdf

MATLAB Release Compatibility
Created with R2007b
Compatible with any release
Platform Compatibility
Windows macOS Linux

Discover Live Editor

Create scripts with code, output, and formatted text in a single executable document.


Learn About Live Editor