removing duplicate data

[EDIT: 20110513 00:34 CDT - reformat - WDR]

I have a text file containing some data and there are some duplicates

line1 : 123 456 789
line2 : 123 456 789
line3 : 234 567 890
line4 : 123 456 789
line5 : 456 789 012

how can I remove the repeated data and save back to my txt file?

Answer 1

[Matt Fig]

Load the data, call the UNIQUE function with the rows option, then save the result.

[Hoa]

I know unique can be used, but I need to compare the 3 different values in every row, that means line1 123 & line2 123, line1 456 & line2 456

and line1 789 & line2 789.

[Matt Fig]

Then you need to be more specific. Given the data:

123 456 789

123 456 789

234 567 890

123 456 789

456 789 012

What do you expect the output to be? Do you want the output to be:

123

456

789

234

567

890

012

or, did you want the output to be:

123 456 789

234 567 890

456 789 012

[Hoa]

yes i mean the second output

[Jan]

Then UNIQUE(Data, 'rows') should work.

[Matt Fig]

fid = fopen('mydata.txt','r');

T = textscan(fid,'%f%f%f','collectoutput',1);T = T{1};

U = unique(T);

fclose(fid);

fid = fopen('mydata.txt','w');

fprintf(fid,'%g\r',U.');

fclose(fid);

[Hoa]

output generated with this is:

line1 1

line2 2

line3 3

line4 4

it output a single digit instead of the whole junk of values

[Walter Roberson]

Please show your current code.

[Hoa]

I used the code by matt fig

[Walter Roberson]

Change the line

fprintf(fid,'%g\r',U.');

to

fprintf(fid,'%g\n',U.');