txt2mat

fast and versatile ascii data import capable of handling large text files

Description

As txt2mat basically is a wrapper for sscanf, it quickly converts ascii files containing m-by-n numeric data, allowing for header lines. When encountering lines with different numbers of data elements, it will work line-by-line and thus slow down somewhat.

You may let txtmat carry out an automatic data layout analysis on comparatively 'simple' text files (header lines + decimal number data with common delimiters). By this analysis it is able to directly import most .csv-files, for instance.

As txt2mat can perform string replacements before the numeric conversion is done, it can cope with many irregularities within the data. By that it is also capable of detecting and handling commas as decimal characters (common german notation).

You can filter lines by keywords, provide appropriate format strings (as for sscanf), or split up the import process for huge files if you encounter memory problems (e.g. a typical regular 40MB file should be imported as a whole within less than 10s on most computers).

txt2mat is intended to work on Matlab R13 and newer versions.

Comments and suggestions welcome.

Andres

• Acknowledgements: This submission has inspired the following:; readMM_2D
• MATLAB release: MATLAB 7.7 (R2008b)

[vincenzo ficco]

 

[Florian H]

Excellent! Finally a function that handles textfiles without the hassle of the builtin MATLAB ones.

I've combined it with
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=15294
to get a really usefull drag-and-drop import funtion for various textformats.

[Wladimir Alonso]

It worked beautifully when I needed to read solar data tables (http://rredc.nrel.gov/solar/old_data/nsrdb/1991-2005/list_by_state.html) that contained headers, text and numbers (in this case using txt2mat(name_of_the_file,1,43,'ConvString',['%d-%d-%d,%d:%f' repmat(',%f',1,38)]) as kindly suggested by the author of the function)
thanks Andres!

[K AM]

Fantastic script. Worked right out of the box. I used it bring in oddly shaped data files and it worked like a dream! Thanks Andres!

[wu zhiyong]

Great work!
Thanks for your suggestion!
It helps me read the data from text files with ignoring the characters.But I fail to ignore the blank lines.
Do you have any suggestion?

[Andres T.]

@wu zhiyong
Please see my posting in the newsgroup.

[rodrigo abarca del rio]

Excellent work. It helped me in reading data from different format, and skipping the headlines ... in just a second, after having tried over more than 4 hours by different methods. we are close to fortran now :-)
thanks so much a lot for your work.

[djr djr]

 

[Ralf]

 

[Zahra]

A powerful code. I am using it (with some help from the author of the code) to read very complicated data files with headerlines throughout the file as well as data lines with different lengths (line folding).

And it is quite fast too.

Thanks Andres for your great work!

Zahra

[John McArthur]

Wondering if there's a way to get the program to recognize and import data in the header line. For instance, if you have the test conditions as header lines, like this:

nTimeInc, 1001, TotalTime, 400
ForcingFreq, 40, ForcingAmp, 15
T, X, Y, Z, Theta, Zeta
0, 0, 0, 0, 0, 1.0
0.4, 1.2, 0, 0, 0.01, 1.2
....

So, the header files have some useful info that would be nice to have accessible and can be added to plots and analysis.

Any thoughts?
[e-mail]

[Andres]

@ John McArthur
The header itself *is* accessible as a string, e.g.

>> [A,ffn,nh,SR,hl] = txt2mat('myfile.txt');
>> headerNumbers = sscanf(hl,'%*s%f,%*s%f')

headerNumbers =
        1001
         400
          40
          15

(generally it is hard to guess which information in the header is useful to the user)

[Fredrik]

 

[Jose Miguel Jauregui]

A great job,
It works perfectly when it comes to load very extensive text files, a very fast code. If you work with extensive text files, this is the code to use.

In case you need a modification Andres is always willing to provide help

Josemi

[Bas]

Needed to have a file import script that could handle a comma as the decimal separator. This worked instantly.

[Gabriel Vézina]

great code and fast to execute

[Val Schmidt]

This seems like a terrifically useful tool.

One feature request. In addition to being able to specify characters that, if found in a line, mark the line for omission, it would be nice to be able to instead skip everything by default and specify characters of lines that are to be included.

Suppose, for example, you want to parse a log file. You'll want to extract all the log entries of a particular type. Rather than having to specify every other type of entry for omission, it'd be nice to be able to specify just the ones you want.

Thanks,
Val

[Leonard]

txt2mat is excellent by being very straightforward in it's implementation. Could be the defacto standard within Matlab. Thanks Andreas for this.

If you are considering compiling a standalone application and deploying it using the MCR, you may want to consider the following:
I did not encounter any errors using this .m file while using it as long as MATLAB was installed on my machine (ver 7.5, BTW).
When deploying my executable to another machine and using the MCR (ver 7.7), the command line indicated the following:

Too many objects requested. Most likely cause is missing [ ] around left hand side that has a comma separated list expansion.

Error in ==> txt2mat at 519

515 %% Definitions
516
517 % find out matlab version as a decimal, up to the second dot:
518 v = ver('matlab');
519 vs= v.Version;
520 vsDotPos = [strfind(vs,'.'), Inf, Inf];
521 vn= str2double(vs(1:min(numel(vs),vsDotPos(2)-1)));

The .m file halted execution of my program because it was looking for a version # for MATLAB, which was not installed on the target machine. I patched the code by determining the matlab ver and editing the txt2mat.m as follows:

518 %v = ver('matlab');
519 vs = 7.5; %vs= v.Version;
520 vsDotPos = [strfind(vs,'.'), Inf, Inf];
521 vn= str2double(vs(1:min(numel(vs),vsDotPos(2)-1)));

Maybe it's possible to check for 'Matlab' on the target machine and/or the MCR and then handle this line appropriately.

Regards,
Len

[Andres]

@ Leonard
Thanks for pointing to the version number issue with the MCR (and that part of code you mention should have been replaced anyhow...). I've just sent you an email regarding a possible solution; if it got lost, please use the 'Contact Author' link on my author page.
Regards
Andres

[Stefanie Peer]

 

[achus Pujante]

Great function, you saved me a lot of time, thank you very much
Then I used your function to simplify my own case

[Pavan]

Superb function. Worked without a problem. Great stuff.
Thanks Andres

[Ana]

 

[Catalin Eberhardt]

I tried using txt2mat to extract a matrix A from a file data.csv that has several lines of explicative text interspersed throughout the file, like this:

text 1
1 2 3
4 5 6
7 8 9
text 2
10 11 12

For the example above, txt2mat produced the matrix

10 11 12

rather than what I expected, which was

1 2 3
4 5 6
7 8 9
10 11 12

Can anyone help? Thanks!

[Catalin Eberhardt]

(I believe a 5 star rating is in order for this function anyway :-) )

[Andres]

It would be most simple if you could uniquely identify every line of text by some string (or some few strings), as with 'tex' (or just 'x') in your example:

txt2mat('example.txt','BadLineString',{'tex'})

If that is not possible, you could read in all lines and then remove all NaN-only rows:

A = txt2mat('example.txt',0,'ReadMode','line')
A = A(any(isfinite(A),2),:)

If things are more complex, you may also contact me via the 'Contact Author' link. Good luck!

[Akshay]

An awesome and VERY helpful function!!! The param/value pairs are great - especially 'BadLineString' and 'RowRange'.

Small suggestion for future spins - if possible, having a 'ColumnRange" would be great.

Btw, I ran into an "Out of Memory" situation when processing a 355MB file, even though I have 3GB RAM and the 'memory' command reported that have atleast ~500MB for a single array (~1200MB for arrays). Any ideas how to fix this?

Thanks!!!

[Andres]

Thanks for your feedback. Indeed it would be nice to have a handy column selection input argument instead of using 'ConvString' with something like '%f %*f %f %*f %f'.
Regarding your memory issue, so far I can only refer to the 'NumericType' param and especially to example 6, but you probably found that in the doc already. txt2mat reads in the whole part of the file that is to be imported and initializes the output array. In an extreme case, a double array holding the data may consume about four times more memory than the file, unless some rows have missing values. I am thinking about making txt2mat more memory efficent - e.g. avoid slurping large files - without noticeably loosing speed (but tbh I have hardly any spare time now).

[Akshay]

Thanks for your reply, Andres. Really, Mathworks should take your function and put it in their base functions - you have taken great efforts!!

[Yvan Lengwiler]

I haven't looked at the code yet, but I have used this submission to read in 1 GB of data. It just worked beautifully, very fast, and allowed me to filter out bogus lines. Very useful submission.

[Matthew]

Fabulously useful code - thanks Andres...

I am particularly impressed with the Param_Array t2mOpts and its' many flexible uses for reading complex data. I hope MATLAB considers standardizing your work.

Also BRAVO with your highly useful documentation!

[arouabm Ben Mohamed]

Hi
thank you very much for this code it is great
but how can we make this function read a text file and convert it into a matrix containing words not characters
actually my file contains strings and numbers

[Andres]

Hi arouabm Ben Mohamed,
I'm sorry, txt2mat does not support mixed string/numeric output in cells as e.g. textscan does. It would be quite hard (probably) to change this without sacrificing speed and R13 compatibility.
Btw, you can use some features of txt2mat (like line filtering) with read mode 'char' and use textscan afterwards (see example 3b).
Perhaps, if ease of use is more important than speed, you may find the fex submission "readtext" useful as well.

[zampala ballarini]

thanks a lot. Have been looking for this for a while.
great job

[Juan Pedro]

Great job

[ESTEBAN]

It works nice and smooth importing DEM (digital elevation models)...Thanks a lot

[Thomas]

I wasted 3 hours searching for a solution to my problem before I stumbled upon this beauty! Thanks so much for all the effort you put into this. Good explanation and examples to get you started. Part of my permanent function collection..

[Henk-Jan Ramaker]

Superb file! Really great! The speed and flexibility is impressive. However, i'm experiencing some troubles. When I read this txt file:

Symbol,Date
XXH00,09/221999

it works great! But, when I read this txt file:

Symbol,Date
ESH00,09/221999

txt2mat cannot read any lines. I discovered, when the 2nd line starts with an 'E' or 'N', txt2mat somehow cannot match the format.

Is there a way around this or to fix this problem?

[Amr Suleiman]