# Filter regions in data with 'obvious' irregular linear trend

2 views (last 30 days)
mashtine on 18 Aug 2015
Commented: mashtine on 20 Aug 2015
Hello,
I have a dataset that I am plotting but there are clear trends occurring within the data that I would like to remove. I have attached a figure of a plot of some of the data. The y variable is supposed to increase exponentially as x increases, but at some points (between 9 and 14 x values), there is a control placed on the y output, causing it to hover around certain values as x increases.
Does anyone know how I may be able to identify these regions where the data has a clear linear trend between certain x values? I would usually think to have a polynominal equation fitted on the data and then index any data points that lie outside the error margins of this fit. However, I can only fit that equation by removing these irregular trends first, so a bit of an issue there.
These linear trend regions do not usually happen in chronological sequence. For instance, there may be a few period in time where the y-output is limited along x, creating a seemingly continuous trend. Thus I cannot use the 'diff' function for instance to easily identify periods when y-output varies by a specified amount. Hope that makes sense.
Feel free to ask me if I need to explain more.

John D'Errico on 18 Aug 2015
Edited: John D'Errico on 18 Aug 2015
This is more difficult to do than you might think. The eye/brain is very good at seeing patterns, even with a lot of noise.
The problem is, this is not a "region",where you have a trend, but a region where SOME subset of the data is spurious, that follows a trend. And it is not even a constant, but an apparently linear trend! I could as easily pick another region wherein some subset of that huge number of points also follows a linear trend, yet they represent perfectly good data in your eyes. What is "obvious" to you is rarely "obvious" to a computer. If that was true, we would all be replaced by computers. (Or is that happening already?)
Sorry, but the best solution might be to use your eye to detect that bad data, perhaps selecting the bad points with the mouse. My selectdata tool on the file exchange might work, although it is getting a bit old. I think it may have some problems with the new graphics engine. There are also brushing options in MATLAB itself now that might suffice for your problem.
mashtine on 20 Aug 2015
haha Will do! Thanks again.