How can I delete the continuously duplicated value and keep the last one in a cell array?

Hello,

I'm trying to delete the continuously duplicated value and keep the last one in a cell array. I try some ways but it does not work. Could you help me correct it?

My input data (around 4 million rows):

    Symbol, Data, Time, Price, Volume
CLF7,01/03/1996,12:54:04,17.58
CLF7,01/03/1996,13:31:19,17.58
CLF7,01/03/1996,15:07:42,17.57
CLF7,01/03/1996,15:07:45,17.58
CLF7,01/03/1996,15:10:03,17.58
CLF7,01/03/1996,12:54:04,17.12
CLF7,01/03/1996,13:27:39,17.58
CLF7,02/03/1996,14:35:40,17.57
CLF7,02/03/1996,12:54:04,17.57

My required output:

    Symbol, Data, Time, Price, Volume
CLF7,01/03/1996,13:31:19,17.58
CLF7,01/03/1996,15:07:42,17.57
CLF7,01/03/1996,15:10:03,17.58
CLF7,01/03/1996,12:54:04,17.12
CLF7,01/03/1996,13:27:39,17.58
CLF7,02/03/1996,12:54:04,17.57

I'm trying with 2 ways. First way, I create a new array with the same size of my data. It will find the value at price (i+1) is the same value at price (i). Then, I will copy the row have price (i+1) into the new array.

     dim = size(data);
data_3 = zeros(dim); 
i = 1; j = 1;
data_3(i,:) = data(i,:); 
for i=2:dim(1)
    if data(i,4) != data(i-1, 4) 
       j = j+1;
       data_3(j,:) = data(i,:); 
    end
end
data_4 = data_3([1:j], :);

With this way, I did not any change in my data. The second way, I use "if" to find the value at a price (i) and compare with price(i-1). If the value is equal, I will delete the row (i-1)

      for i=2:size(data);
If data(i,4) ~ data(i-1,4);
data(i-1,4)=[];
end
end

However, I meet the error: "Conversion to logical from struct is not possible".

I'm tried to find some similar questions in forum, but it is almost found and deleted all duplicate value compared the first appeared value and keep the last one. For my requirement, I just found and deleted the duplicated value when it is continuously duplicated value.

I'm really appreciated your help.

Thanks

Answer 1

[Andrei Bobrov]

f = fopen('20160609.txt');
c = textscan(f,'%s %s %s %f','delimiter',',');
fclose(f);
t =[diff(c{end})~=0;true];
C = [c{1:3}];
out = [C(t,:),num2cell(c{end}(t))];

[Trung Hieu Le]

Thanks a lot. Your code work perfect. However, can you recommend the code to load multiple files? Because I have a lot of files like that. I tried to edit as follows but it did not work.

    for i=1996:1:1999;
  fID=fopen({['CLF',num2str(k),'.txt']});
    Ci=textscan(fID, '%s %s %s %f %f','Headerlines',1,'delimiter',',');
    fclose(fID);
t =[diff(fID{4})~=0;true];
C = [fID{1:3}];
out = [C(t,:),num2cell(fID{4}(t))];
end

In addition, I tried to combine the below code to delete the value 0 and NaN at price column, but I failed with this code

    for i=1996:1:1999;
  fID=fopen({['CLF',num2str(k),'.txt']});
    Ci=textscan(fID, '%s %s %s %f %f','Headerlines',1,'delimiter',',');
    fclose(fID);
fID(isnan(fID(:,4)==1)=[];
fID(fID(:,4)==0)=[];
t =[diff(fID{4})~=0;true];
C = [fID{1:3}];
out = [C(t,:),num2cell(fID{4}(t))];
end

Could you help me take a look about my code? Attached is my file (just disregard the column 5, because I want to delete it. Fortunately, When I edit and run your code, it has deleted automatically)

I want to express my unlimited gratitude.

[Andrei Bobrov]

f = fopen('20160610.txt');
c = textscan(f,'%s %s %s %f %f','delimiter',',','HeaderLines',1,...
                                                         'EmptyValue',nan);
fclose(f);

C = [c{1:3}];
t = ~isnan(c{4});
C = C(t,:);
co = c{4}(t);
to =[diff(co)~=0;true];
out = [C(to,:),num2cell(co(to))]

[Trung Hieu Le]

Great work. Many thanks a lot for your help. I will do this code for each file. Thanks again