Reading text file word by word

Asked by Paolo Binetti on 13 Oct 2018

Latest activity Edited by Paolo Binetti on 13 Oct 2018

184 views (last 30 days)

dataset_300_8.txt

Input is the attached text file, with one long word, a newline, then several equal-length short words separated by white spaces. I would like to read the first word in a variable, then the other ones in another variable one by one, rather than all into a single string or into a single huge cell array. I tried to do this with fscanf in several ways, but failed, and even got the impression that fscanf is not complying with https://fr.mathworks.com/help/matlab/ref/fscanf.html no clue about what I am doing wrong.

fileID = fopen('dataset_300_8.txt');
long_word = fscanf(fileID, '%[$ACGT]'); % is there another way to stop reading at newline? 
short_word = ' ';
while ~isempty(short_word)
  short_word = fscanf(fileID, '%s'); % does not work: shouldn't %s stop as it encounters a white space?
  % short_word = fscanf(fileID, '%10s'); % this also does not work
  % short_word processing code here
end
fclose(fileID);

4 Comments

Show 1 older comment

jonas on 13 Oct 2018

I would just read the entire thing and split it, but I do not know if that satisfies the ' read one by one' requirement.

madhan ravi on 13 Oct 2018

%10c and the delimiter does the work as my answer below.

Paolo Binetti on 13 Oct 2018

Thank you but reading all the content of the file in one shot is what I want to avoid. The "file" variable from Madhan's code has 127134 Bytes. What I want is a variable long_word of 2002 Bytes plus a variable short_word of 20 Bytes, into which I want to read each of the short words in the text file one by one. I read one, I use it for computations, then I don't need it anymore, so I read the next one, and so forth. I want to get the job done with 2022 Bytes, rather than 127134 Bytes. My file is just a small sample, but for bigger files memory would be an issue.

Products

MATLAB

Release

R2018b

2 Answers

Link

Answer by jonas on 13 Oct 2018

Edited by jonas on 13 Oct 2018

Accepted Answer

Try this minor change

short_word = fscanf(fileID, '%s+');
                               ↑

Edit: After further testing, any character after the %s gives the same results as it causes the fscanf to stop reading (due to mismatch). Another iteration begins where the previous attempt failed, so at the next word.

6 Comments

Show 3 older comments

Paolo Binetti on 13 Oct 2018

Thank you, it works. It is about 10x slower than the vectorized operation, but it satisfies my memory requirement.

Yet, what does the "+" mean? Its use for fscanf seems undocumented here https://fr.mathworks.com/help/matlab/ref/fscanf.html

And what was I doing wrong, like if I was telling scanf to read 10 characters at a time (%10s), why was it not doing it?

jonas on 13 Oct 2018

+ just means to continue reading characters until something else is encountered. It is used in textscan so I just assumed it applies here as well.

"A = fscanf(fileID,formatSpec) reads data from an open text file into column vector A and interprets values in the file according to the format specified by formatSpec. The fscanf function reapplies the format throughout the entire file and positions the file pointer at the end-of-file marker. If fscanf cannot match formatSpec to the data, it reads only the portion that matches and stops processing.

So formatSpec (%s) reads all characters, skips whitespaces and returns a single long character sequence whereas (%c) does the same but retains the whitespaces.

This means that adding any character after (%s) forces the scan to stop processing and the pointer is placed where the scan failed due to mismatch. If you do another fscan, then it continues to read where failed previously.

Paolo Binetti on 13 Oct 2018

Clear, thank you

Link

Answer by Image Analyst on 13 Oct 2018

"I would like to read the first word in a variable, then the other ones in another variable one by one, rather than all into a single string or into a single huge cell array." <--- this is a really bad idea. I'm sure Stephen will soon give you the reasons why.

Better solution is to use fileread() followed by strsplit() to make the single cell array.

str = fileread('dataset_300_8.txt'); % Read entire file.
ca = strsplit(str, ' '); % Put each word into a cell

1 Comment

Paolo Binetti on 13 Oct 2018

Really wonder why. OK, what if the combined size of the small words is 100 GB: guess you can't throw all of it in a single variable right? The file is on the hard disk, which can handle big chunks of data, but once you read it into a variable it goes into RAM, which cannot, right? So the idea was to read a piece of data at a time, process it, then read the next one, process it, etc.

Products

MATLAB

Release

R2018b

You are now following this question

Paolo Binetti (view profile)