How to retrend data, after detrending it?

I have some data (inputClosePrice)

i log it, detrend it, and then normalise it.

How can i now get back to the intial set of data for a given point?

detrend function:

https://au.mathworks.com/help/ident/ref/iddata.detrend.html

I've had a look at the retrend function,however, in the example they talk abotu iddata, linear ARX model, simulate model, and i do none of that, as part of my detrending process, so i'm unsure what to send the retrend function.

https://au.mathworks.com/help/ident/ref/iddata.retrend.html

--- sample of my code given below:

sdata_log = log10(inputClosePrice); %log the data
detrend_LogPrice = detrend(sdata_log, inputPolyDegree); %detrended log price
normalised_detrendedData = detrend_LogPrice;

Answer 1

[Star Strider]

There are three different ddetrend funcitons. The one you want is: detrend however it does not return the function it used to detrend the data. If you want to retain that function, use polyfit and polyval instead to create and evaluate the detrending polynomial, then subtract it from the data, do whatever analysis you want, then add that polynomial back to the processed data.

[Rizwan Khan]

Dear Sir,

Can you tell me the other detrend functions?

I make use of polyfit and polyval separately, but knowing the value of the polyval does not give the value of the actual data, because there is significant error between the polynomial value and the actualy data.

There is a second way i approach this this problem as well, which I've pasted below.

Here i create the linear regression model fit, i then normalise the residuals (between 0-1)

Now what i want to know if what does the "price_log10" need to be if the normalised residual is 0.1

So my guess is that first i need to remove normalisation, to get back to what the residual value woudl be.

Then i need to maybe add that to the mdl or something, to get back to actual value

Your thoughts?

%do linear regression model on it, and normalise the residuals 
        mdl = fitlm(date_log10,price_log10, "linear");
        normalised_detrendedData = normalize(mdl.Residuals.Raw, 'range');

[Star Strider]

I do not understand tha problem, so I have no idea how to reply.

[Rizwan Khan]

Dear Sir,

Let me try to better explain the problem

I have some data

i detrend the data using a regression model (fitlm) and obtain the residuals.

mdl = fitlm(date_log10,price_log10, "linear"); %this gives me the regression model

mdl.Residuals.Raw %these are the residuals from the regression model

i then normalise the residuals, so they are simply between 0-1

normalised_detrendedData = normalize(mdl.Residuals.Raw, 'range');

So some residuals may have value of 0.1, others a value of 0.3 and so on.

I would like to find out, what is the initial data value for a given residual

For eg. if the residual has a value of 0.3, what is the data value at that point?

My guess is i need to reverse the order of operations

so i need to first somehow reverse the normalisation to get back to the residual.

then i need to maybe add the residual to the MDL (regression model) to get back to the original data value?

Does that better explain it?

[Star Strider]

I do not understand the reason for normalising the residuals. The residuals themselves would not appear to be important. The entire purpoose of detrending is to remove a linear or polynomial trend from a recorded signal, something that can be done by simple regression. Then to reconstitute the processed signal, simply add the removed trend back.

I agree that reversing the order of operations is important if the intent is to reconstitute the original signal baseline trend after processing it.

[Rizwan Khan]

Hi Strider,

I normalisze the residuals so i can make the color graph below so the normalised residuals are sent into the scatter plot.

You agree on reversing the order of operation to reconstiture the original data, however, how do i do that?

[Star Strider]

Please share the data used to produce that plot, and a detailed description of what you want to do with it.