How to segment a curve into linear and nonlinear pieces

I have a dataset which looks linear at the beginning (up to ~3.5 along x-axis) and then it shows and upward climb. I want to locate an ideal point uptill where the data is linear at the beginning and fit a line. Thanks a lot. The data is attached (1st column is X and 2nd column is Y).

[Bruno Luong]

linear about 3.5 ? When I zoom in it is not clear where the linear starts. Not 3.5, not 2, may be on the interval (0,1)

[Ahmed Zohaib]

Thanks for the feedback. Okay, how do I locate the exact point where it loses linearity in between [0 1]?

Answer 1

[Chunru]

load(websave("DatasetXY.mat", "https://www.mathworks.com/matlabcentral/answers/uploaded_files/1437433/DatasetXY.mat"));
whos

  Name            Size             Bytes  Class     Attributes

  XY          52651x2             842416  double              
  cmdout          1x33                66  char                

[~, ia] = unique(XY(:, 1));   % remove the duplicated points
XY = XY(ia, :);
plot(XY(:,1), XY(:,2));
gg = gradient(gradient(XY(:, 2), XY(:, 1)), XY(:, 1));  % 2nd derivative
gg = movmedian(gg, 20);      % to remove some spikes
hold on;

whos

  Name          Size            Bytes  Class     Attributes

  XY          818x2             13088  double              
  cmdout        1x33               66  char                
  gg          818x1              6544  double              
  ia          818x1              6544  double              

% ignore some initial points
ip = 20;        % 20 points
gg((ip+1):end)

ans = 798×1

0 0 0 0 0 0 0 0 0 0

i = find(abs(gg((ip+1):end))>20, 1, 'first');      % 2000 as a threshod (adjust it)
i = ip+i

i = 301

plot(XY(i,1), XY(i, 2), 'ro');

%xlim([2 3])

Answer 2

[Bruno Luong]

Ths produces kind of "arbitrary" choice of the breakpoint, consider fitting the global curve with few line segments

load(websave("DatasetXY.mat", "https://www.mathworks.com/matlabcentral/answers/uploaded_files/1437433/DatasetXY.mat"));

X=XY(:,1);
Y=XY(:,2);
% FEX https://uk.mathworks.com/matlabcentral/fileexchange/25872-free-knot-spline-approximation
% it takes a dozain of seconds here
pp = BSFK(X,Y,2,[],[],struct('Animation',true))
xbreaks=pp.breaks(2)
ybreaks=ppval(pp,xbreaks)

plot(xbreaks,ybreaks,'or','MarkerSize',15)

[Bruno Luong]

If I crop the data up to X <= 4 it finds the break at

xbreaks=2.3085

As I said it's a mmoving target. The data is more or less exponetial everywhere. It is just a question of scaling, totally subjective.

load(websave("DatasetXY.mat", "https://www.mathworks.com/matlabcentral/answers/uploaded_files/1437433/DatasetXY.mat"));
X=XY(:,1);
Y=XY(:,2);

b = X < 4
% FEX https://uk.mathworks.com/matlabcentral/fileexchange/25872-free-knot-spline-approximation
% it takes a dozain of seconds here

pp = BSFK(X(b),Y(b),2,[],[],struct('Animation',true))
xbreaks=pp.breaks(2)
ybreaks=ppval(pp,xbreaks)
plot(xbreaks,ybreaks,'or','MarkerSize',15)