Rank: 187 based on 385 downloads (last 30 days) and 11 files submitted
photo

M Sohrabinia

E-mail
Company/University
University of Canterbury
Lat/Long
-43.523716, 172.5843

Personal Profile:
Professional Interests:
Image processing of the remotely sensed data

 

Watch this Author's files

 

Files Posted by M View all
Updated   File Tags Downloads
(last 30 days)
Comments Rating
10 Sep 2014 Screenshot Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia outliers, thompson tau, matrix outliers, indexes of outliers 100 9
  • 3.33333
3.3 | 6 ratings
06 Jun 2013 Screenshot Find commontimes Finds common date-times from two or more date-time vectors with a given tolerance in seconds Author: M Sohrabinia find common times, common dates, common datetimes, similar times, similar dates, matching times 8 0
22 May 2013 Screenshot Read netCDF files Reads netCDF data for given fields from a list of given files Author: M Sohrabinia read netcdf files, read nc fields, timeseries from netcd..., read netcdf file attr..., extract netcdf fields 16 0
14 Feb 2013 Screenshot Build custom colormaps Buillds any sequence of colormaps based on wrgbcmyk colors Author: M Sohrabinia colormap, custom colormap, build colormap, color gradient, bw colormap, rgb colormap 76 3
  • 4.5
4.5 | 2 ratings
10 Dec 2012 Screenshot Atmospheric pressure unit conversion A simple tool to be integrated in longer scripts which require pressure conversions Author: M Sohrabinia atmospheric pressure ..., pa to psi, pa to bar, pa to atm, pa to torr 11 0
Comments and Ratings by M View all
Updated File Comments Rating
14 Sep 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia

@Rob: did you try the updated code?

10 Sep 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia

@John D'Errico: Thanks for your valuable feedback, it was very detailed and helpful, like those comments that sometimes I get from good reviewers. However, I do not agree with your point about Jered's comment (what you interpret from that comment is not what I read from it unless there is a magic meaning there that I dont get it). Anyway, I have put more notes in the code which clarifies that this code won't be able to find an outliers on a nicely fitted curve. I have another code submitted here named 'regoutleirs' which might be useful to find outliers in bivariate fitted data. It uses the residuals vector.Thanks again for your useful feedback

23 Aug 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia

@Jered: I wonder why you are commenting and rating without understanding the code properly, just take a look at the examples given above, do they have zero mean? What were your errors using this code? You need to be specific if you really want to provide useful feedback.

01 Dec 2013 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia

@Ahmad: At this stage, my function can only handle a column/row vector or a matrix but you can call this function three times each time giving one matrix of your 3D matrix, it will remove outliers from that matrix according to Thompson Tau rule. If you need to remove the outliers from the 3D matrix considering all values in one go, you would need to concatenate the 3D matrix into a single matrix, call this function and then reconstruct the 3D matrix in the same order you had concatenated.

27 Sep 2013 Build custom colormaps Buillds any sequence of colormaps based on wrgbcmyk colors Author: M Sohrabinia

@Alfredo: thanks for your comment, you mean the range of the output? such a 0 to 1?

Comments and Ratings on M's Files View all
Updated File Comment by Comments Rating
14 Sep 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia Sohrabinia, M

@Rob: did you try the updated code?

14 Sep 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia Campbell, Rob

In John's first example, there is no "nicely fitted curve" yet the outlier is not spotted.

10 Sep 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia Sohrabinia, M

@John D'Errico: Thanks for your valuable feedback, it was very detailed and helpful, like those comments that sometimes I get from good reviewers. However, I do not agree with your point about Jered's comment (what you interpret from that comment is not what I read from it unless there is a magic meaning there that I dont get it). Anyway, I have put more notes in the code which clarifies that this code won't be able to find an outliers on a nicely fitted curve. I have another code submitted here named 'regoutleirs' which might be useful to find outliers in bivariate fitted data. It uses the residuals vector.Thanks again for your useful feedback

23 Aug 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia D'Errico, John

What Jered is trying to point is indeed a serious flaw in this code.

Consider these two examples. In the first case, I'll create an outlier that has large absolute value.

x = rand(10,1) + 10;
x(3) = 20;
y = outliers(x)
y =
10.439
10.382
NaN
10.795
10.187
10.49
10.446
10.646
10.709
10.755

outliers finds the element that is inconsistent with the data, and replaces it with nan. No problem there.

However, suppose we create a point that is just as much of an outlier? (In fact, it is more significantly an outlier by some measures, if we compare it to the mean of the remainder of the data.)

x(3) = 0;
y = outliers(x)
y =
10.439
10.382
0
10.795
10.187
10.49
10.446
10.646
10.709
10.755

See that outliers fails to identify that point.

The problem is that this code does indeed look at the simple magnitude of the point. Points with large magnitude are flagged as an outlier.

Another example might be appropriate. I'll create a nice smooth curve here.

t = linspace(0,2*pi,21)';
x = sin(t);
x(11)
ans =
1.2246e-16

So the 11th point is zero. (essentially.) I'll change it to 1, a value that is clearly not on the nice smooth curve.

x(11) = 1;
y = outliers(x)
y =
0
0.30902
0.58779
0.80902
0.95106
1
0.95106
0.80902
0.58779
0.30902
1
-0.30902
-0.58779
-0.80902
-0.95106
-1
-0.95106
-0.80902
-0.58779
-0.30902
-2.4493e-16

Oops, outliers fails to find a point that is obviously not on the curve. I'll make the outlier more obvious this next time. Clearly x here will be always positive, but I'll make one element fairly clearly an outlier, thus completely inconsistent with the remainder of the curve.

x = sin(linspace(0,2*pi,10)')+2;
x(5) = -2
x =
2
2.6428
2.9848
2.866
-2
1.658
1.134
1.0152
1.3572
2

y = outliers(x)
y =
2
2.6428
2.9848
2.866
-2
1.658
1.134
1.0152
1.3572
2

Again, outliers fails to find that point.

Jered has pointed out that this tool assumes the data is centered around zero, then it looks for points that have large absolute magnitude, and discards those points.

So while I like SOME ASPECTS about the documentation in this tool, and I like the way the code was written, the tool simply has a major flaw. I've not looked at the references, so I do not know if the basic underlying algorithm is just poor, or if the implementation is poor. This code fails to find outliers in your data, UNLESS those outliers are of a very specific class, thus large in magnitude compared to the rest of the data. The code fails to use any information about whether the curve is smooth or noisy, it merely looks for points that are large in absolute value compared to the rest.

There are other problems with this tool. The author has put a blank line into the code after the fist line. So despite the voluminous set of help written, when you use help, you get essentially no help at all.

>> help outliers
outliers function: remove outliers based on Thompson Tau:

>>

Help looks for a CONTIGUOUS block of comments, and dumps that block out to the command line. A blank line in the middle cuts help off at the knees.

I wanted to give this code 3 stars because the author made some effort despite the MAJOR flaw with the algorithm, but the lack of usable help reduces my rating to 2 stars.

To the author, don't tell me I don't understand the code.

To any potential users, I would suggest you do as Jered said - be very careful about using this code. Code that finds only a very restricted class of outlier is not good at all, unless the only outliers you will ever see are those it can find.

23 Aug 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia Sohrabinia, M

@Jered: I wonder why you are commenting and rating without understanding the code properly, just take a look at the examples given above, do they have zero mean? What were your errors using this code? You need to be specific if you really want to provide useful feedback.

Contact us