MATLAB Answers

getting a NaN in correlation coefficient

562 views (last 30 days)
Sumera Yamin
Sumera Yamin on 19 Feb 2020
Commented: Adam Danz on 18 Mar 2021
Hi, i have a simple problem which unfortunately i am unable to understand.
I have matrices and i am trying to calculate correlation coefficient between two variables. A simple example from my code is attched. Why am i getting a NaN here. What does this implies
x=[-7.501899598769999514e-04;-6.501899598769999514e-04;-5.501899598769999514e-04];
y=[-0.414;-0.414;-0.414];
c11=corr2(x,y)

Accepted Answer

Adam Danz
Adam Danz on 19 Feb 2020
Edited: Adam Danz on 27 Aug 2020
When NaNs appear in the output but are not present in the inputs
Notice that all of the values in y are identical y=[-0.414; -0.414; -0.414];
If you look at the equations for corr2() or Pearson's corr() you'll notice that both have a term in the denominator that subtracts the mean of y from each y-value. When each value of y is identical, the result is a vector of 0s. When you divide by zero, you get NaN.
Another way of putting it, the standard deviation of x or y cannot be 0. When you have a vector of identical values, the std is 0.
The NaN, in this case, is interpretted as no correlation between the two variables. The correlation describes how much one variable changes as the other variable changes. That requires both variables to change.
NaN values in the inputs spreading to the outputs
For r=corr2(x,y):
When there is 1 or more NaN values in the inputs, to corr2(x,y), the output will be NaN. Fill in the missing data before computing the 2D correlation coefficient.
For r=corr(x):
A single NaN value in position (i,j) of input matrix x will result in a full row of NaN values at row i and a full column of NaN values in column j of the output matrix r (see explanation).
x = [
6 5 1
3 NaN 9
5 3 7
9 5 5 ];
r = corr(x)
1 NaN -0.52699
NaN NaN NaN
-0.52699 NaN 1
For r=corr(x,y):
A single NaN value in position (i,j) of either x or y inputs will results in a column of NaN values in column j of the output matrix r.
x = [
9 5 1
1 4 4
2 6 4
2 5 9 ];
y = [
6 5 1
3 NaN 9
5 3 7
9 5 5 ];
r = corr(x,y)
0.1623 NaN -0.92394
0.3266 NaN -0.23905
0.62312 NaN 0.32367
Ignoring NaNs in corr() inputs
The rows option in corr() can be set to complete or pairwise which will ignore NaN values using different methods.
'rows','complete' removes the entire row if the row contains a NaN. In other words, it will remove row 2 from both x and y input matrices. Using the same inputs above,
r = corr(x,y,'rows','complete')
-0.27735 0.5 -0.94491
-0.69338 -1 0.75593
0.81224 0.14286 0.53995
r2 = corr(x,y) % for comparison
0.1623 NaN -0.92394
0.3266 NaN -0.23905
0.62312 NaN 0.32367
Notice that this changes all of the correlation values since the entire row #2 was removed from both inputs x and y. To confirm that, we can remove those rows and recompute the correlation matrix.
% Remove row 2 which contains a NaN in y
r3 = corr(x([1,3,4],:) ,y([1,3,4],:));
-0.27735 0.5 -0.94491
-0.69338 -1 0.75593
0.81224 0.14286 0.53995
Voila! Outputs r and r3 match.
'rows','pairwise' only removes rows only if a NaN appears in the pairing of two columns. For the same x, y inputs as above, the correlation with columns in x paired with the 2nd column in y will omit the NaN and will be based on the remaining 3 values. All other column-paired correlations will use all 4 rows of values.
r = corr(x,y,'rows','pairwise')
0.1623 0.5 -0.92394
0.3266 -1 -0.23905
0.62312 0.14286 0.32367
r2 = corr(x,y) % for comparison
0.1623 NaN -0.92394
0.3266 NaN -0.23905
0.62312 NaN 0.32367
Notice that values in columns 1 and 3 haven't changed since they do not involve column #2 in y. To confirm the correlation values in column 2 of r,
% Remove row 2 which contains a NaN in y
r3 = corr(x([1,3,4],:) ,y([1,3,4],:));
% Replace NaN column in r2 with new r values
r2(:,2) = r3(:,2)
0.1623 0.5 -0.92394
0.3266 -1 -0.23905
0.62312 0.14286 0.32367
Voila! Updated output r2 matches r.
  10 Comments
Sumera Yamin
Sumera Yamin on 28 Aug 2020
many thanks for this excellent explaination. I understand it very well now.

Sign in to comment.

More Answers (1)

Jan Pokorny
Jan Pokorny on 18 Mar 2021
Hi, I tried this funkcion as bellow, but it returns all NaN. I don't know why, any idea?
I then lately solved this by making cov() and then corrcov(), but it shouldn't go like that.
Data=[3 0 2 0 3 11 7 6 6 25 31 22 25 48 109 85 67 110 205 124 158 114 126 185 291 259 373 262 160 184 307 281 269 332 282 115 235 195 295];
for Zp=0:21
X=Data(1,1+Zp:39);
Y=Data(1,1:39-Zp);
RMat=corr(X,Y);
Koef(1+Zp)=RMat(1,2);
end
plot(0:Zp,Koef,'red*-')
  1 Comment
Adam Danz
Adam Danz on 18 Mar 2021
From the documentation,
rho = corr(X,Y) returns a matrix of the pairwise correlation coefficient
between each pair of columns in the input matrices X and Y.
Note the word columns. You're using rows.
Instead,
RMat=corr(X(:),Y(:))

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!