View License

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video

Highlights from
LOWESS, Locally Weighted Scatterplot Smoothing for linear and non-linear data (enhanced)

4.8 | 11 ratings Rate this file 35 Downloads (last 30 days) File Size: 41.9 KB File ID: #22470 Version: 1.4
image thumbnail

LOWESS, Locally Weighted Scatterplot Smoothing for linear and non-linear data (enhanced)


Jeff Burkey (view profile)


16 Dec 2008 (Updated )

LOWESS, a robust regression like LOWESS allows detecton of a trend otherwise with too much variance

| Watch this File

File Information

LOWESS- Locally Weighted Scatterplot Smoothing that does not require the statistical toolbox in matlab.

This regression will work on linear and non-linear relationships between X and Y.

12/19/2008 - added upper and lower LOWESS smooths. These additional smooths show how the distribution of Y varies with X. These smooths are simply LOWESS applied to the positive and negative residuals separately, then added to the original lowess of the data. The same smoothing factor is applied to both the upper and lower limits.

 2/21/2009 - added sorting to the function, data no longer need to be sorted. Also added a routine such that if a user also supplies a second dataset, linear interpolations are done one the lowess and used to predict y-values for the supplied x-values.

10/27/2009 - modified the second user provided X-data for obtaining predictions. Matlab function unique sorts by default. It really was not needed in the section of code to perform linear interpolations of the x-data using the y-predicted LOWESS results. If the user does not supply a second x-data set, it will assume to use the supplied x-y data set. Thus there is an output (xy) that maintains the original sequence of the input. Additionally, the user can now include a sequence index as the first column of input data. This can be a datenum or some other ordering index. The output will be sequenced using that index. If a sequence index is provided a second subplot will be created show the predicted Y-values in the order of the included sequence index. I suspect this sequence index most often will be a DateTime (i.e. datenum). Just to the function generic enough, the X-axis labels are not converted to a nice date format, but the user could easily change that with a datetic attribute in the subplot.

6/15/2012 - oddly, when using this routine on data without a time sequence (i.e. a third column), the plotting portions cause an error. Not sure how I would have missed that but...I think I have it fixed.

Using a robust regression like LOWESS allows one the ability to detect a trend in data that may otherwise have too much variance resulting in non-significance p-values.

Yhat (prediction) is computed from a weghted least squares regression whose weights are both a function of distance from X and magnitude from of the residual from the previous regression.

The logic of these functions and subfunctions follow the USGS
Kendall.exe routines. Because matlab is 8-byte precision, there are some very small differences between FORTRAN compiled and matlab. Maybe 64-bit OS's has 16-byte precision in matlab?

Data are expected to be sorted prior to data input for this function. Sorted on first column of datain.

There is a very simple subfucntion to create a plot of the data and regression if the user so choses with a flag in the call to the lowess function. BTW-- the png file looks much better than what the figure looks like on screen.

There are loops in these routines to keep the memory requirements to a minimum, since it is foreseeable that one may have very large datasets to work with.

f = a smoothing factor between 0 and 1. The closer to one, the more smoothing done.

   [dataout lowerLimit upperLimit]
                    = lowess(datain,f,wantplot,imagefile)

  datain = n x 2 matrix
  dataout = n x 3 matrix
  wantplot = scaler (optional)
       if ~= 0 then create plot
  imagefile = full path and file name where to output the figure to an
       png file type at 600 dpi.
       e.g. imagefile = 'd:\temp\lowess.png';

  datain(:,1) = x
  datain(:,2) = y
  f = scaler (0 < f < 1)
  wantplot = scaler
  imagefile = string

 datain must be sorted prior to loading into this function on the
 x-value. This is not done in the function because the user may want to have the end result be unsorted (e.g. time sort).

dataout(:,1) = x
dataout(:,2) = y
dataout(:,3) = y-prediction (aka yhat)
lowerLimit(:,1) = x with negative residuals
lowerLimit(:,2) = y-prediction of residuals + original y-prediction
upperLimit(:,1) = x with positive residuals
upperLimit(:,2) = y-prediction of residuals + original y-prediction

Requirements: none

Written by
Jeff Burkey
King County Department of Natural Resources and Parks

MATLAB release MATLAB 7.7 (R2008b)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (18)
09 Jan 2017 Muhammad Haziq Kamarul Azman

Sorry, the equation should've been:

b = (length(x)*wxy-wx*wy)/(length(x)*wxx-wx^2);

Comment only
09 Jan 2017 Muhammad Haziq Kamarul Azman

Very good piece of code.

Just one question though. Why is it, in the function wlsq, that the calculation of b is missing the signal length N? Shouldn't it be:

b = (wxy-wx*wy)/(length(x)*wxx-wx^2);

Comment only
04 Jan 2017 Szymon Beczkowski

Very verbose function. Remove all unnecessary fprintfs and replace the necessary ones with warnings. Do not put tic/toc inside the function.

21 Oct 2016 Ji

Ji (view profile)

06 May 2016 Alex

Alex (view profile)

does anyone know whether this uses linear regression, or polynomial regression with degree 2 to do the smoothing? Is it possible to modify it to use weighted moving averages, i.e. only fit the constant term in regression?

30 Oct 2015 Carl Witthoft

Recommend input validation for "f". E.g.,

if ( ~exist('f','var') || f <=0 ),
f = 0.1;

And it definitely needs to allow x=0. Imagine a NASA launch timeline, "T minus 5, Tminus 4, 3,2,1,...Oh wait, we can't allow a zero...T plus 1..." :-)

06 Jul 2015 Mike Bindschadler

Thanks for a great function which generally appears to work very well. However, users should be aware that any data point with an x value of zero will be ignored. It's not clear what the reasoning for this is, but I had to change it to treat points with x values of 0 the same as any other point.

Comment only
21 Jan 2015 Michael Schwartz

Works great - just as described. Well documented. Thanks a lot Jeff.

15 Jun 2012 Yan

Yan (view profile)

Say at time point 0, we have measured the concentration of a molecule. What measured is just the initial value.

Comment only
15 Jun 2012 Yan

Yan (view profile)

Many thanks to Jeff for replying me and solving the problem.

15 Jun 2012 Jeff Burkey

Jeff Burkey (view profile)

If one truly measures zero, what would be the expected response? A constant? I can see how that may pose a problem, but what is the question imposed on the data? This is a good question. I did not consider measuring zero. Give me a few days to revisit and have a response. - Jeff

Comment only
15 Jun 2012 Yan

Yan (view profile)

If datain is an n×2 matrix, the first column of datain does not allow 0.

Comment only
15 Jun 2012 Yan

Yan (view profile)

I tried the function, and it worked for part of my problems. It seems that the code does not support if datain contains 0 in it? If there is 0 in the datain, the error report is:

??? Undefined function or variable "xy".

Error in ==> lowess at 269
customplot(dataout,upperLimit,lowerLimit,f,[dte x_data

31 Mar 2011 Chih-Wei Tsai

05 Feb 2010 Andre Guy Tranquille

05 Aug 2009 Jeff Burkey

Jeff Burkey (view profile)

You shouldn’t need to modify the function to not input the “xdata”. If you don’t input it, the function will perform just not that task. If that is the case, don’t specify the “xy” as part of your output either.

The “dataout” variable contains the predicted y for each x- value supplied in the “datain” variable. The “xdata” is if you want a different set of y-predictions using the lowess function for other x-data not provided in the “datain” variable.

I don’t remember if the function I posted sorts or not, but if you are wanting to regress as a function of time, you may not want to sort.

I hope this helps.

- Jeff

Comment only
05 Aug 2009 Kaite

Kaite (view profile)

awsome! This is exactly what I need. I am a little confused about one thing...I have an n x 2 matrix in datain that consists of x=precip and y=groundwater level that I would like to do the regression on and get the y-predictions using the lowess regression results. I modified the function to not include the "xdata" and "xy" but I am getting errors about the try-catch syntax on line 298 of the m-file.I have version 7.4.0 (R2007a). The error states that it will continue to run but I dont get results. Am I putting the data in the wrong place? Do I need to put a second data set in? I wanted to take the resulting predicted y values and run the Mann-Kendall test to see if i see a significant trend. what is the purpose of the second dataset of x-values? As you can tell I am new to Matlab and it would be wonderful if I could figure this out! thanks so much!! If i can get this to work for my data this would be AMAZING!

21 Jun 2009 Jeff Evans

Fantastic! Worked perfectly the first time. I did modify it just a bit to fit my purposes by suppressing the screen output and only returning the dataout variable, since don't want to plot the confidence limits. Thanks for the contribution.

- Jeff

19 Dec 2008 1.1

Added computation of upper and lower smooths of the residuals, also will plot on figure.

21 Feb 2009 1.2

Added a sorting call. Also added a few lines of code allow the user to supply a second set of x-values to be used for y-predictions using the results of the lowess regression.

27 Oct 2009 1.3

Updated graphing. Revised input. Includes example data file. Revised output.

18 Jun 2012 1.4

There wasnt really an issue having zero values; rather, when supplying data with no time sequence, the interpolation would error.

Contact us