Code covered by the BSD License  

Highlights from
txt2mat

4.86111

4.9 | 36 ratings Rate this file 89 Downloads (last 30 days) File Size: 30.1 KB File ID: #18430

txt2mat

by

 

23 Jan 2008 (Updated )

fast and versatile ascii data import capable of handling large text files

| Watch this File

File Information
Description

As txt2mat basically is a wrapper for sscanf, it quickly converts ascii files containing m-by-n numeric data, allowing for header lines. When encountering rows with different numbers of data elements, it will work line-by-line and thus slow down somewhat.
You may let txtmat carry out an automatic data layout analysis on comparatively 'simple' text files (header lines + decimal number data with common delimiters). By this analysis it is able to directly import most numeric .csv-files, for instance.

As txt2mat can perform string and regular expression replacements before the numeric conversion, it can cope with many irregularities within the data. By that it is also capable of detecting and handling commas as decimal characters (common german notation).

You can filter lines by keywords, skip lines by line number, provide appropriate format strings (as for sscanf), or split up the import process for huge files if you encounter memory problems.

You may also use the above to simply read the manipulated text into a string or to put each line into a separate cell without the numeric conversion.

txt2mat should work on Matlab R2007a and newer versions.

Comments and suggestions welcome.

Andres

Acknowledgements

This file inspired Read Mm 2 D.

MATLAB release MATLAB 8.3 (R2014a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (49)
06 May 2014 Tom Are  
15 Apr 2014 Marcelo Toledo  
23 Mar 2014 Andres

Hi Kaare, I had no success in contacting you via your author page, but thanks for your suggestion. And yes, a new version of txt2mat capable of skipping lines by line number will come up soon.

23 Jan 2014 Kaare

Very good :) Very fast!
Would it be possible to also include a "skip every n'th line" argument? It's really all it needs for me to discard my own implementation and use this instead.

12 Sep 2013 Andre

Great! Thanks a lot. Works perfect!

07 Jun 2013 Liping LI

perfect~~

05 Oct 2012 Jânio Anselmo

Perfect!!

15 Aug 2012 Carl Fischer

Great. Thanks a lot.

10 Jul 2012 Kyle

Brilliant work - saved me a day. Much better than Matlab itself.

26 May 2012 Clemens

Great work! Your submission provides a effortless solution which is just doing what I was expecting in every case I've used it.

01 Feb 2012 Amr Suleiman  
12 Jan 2012 Henk-Jan Ramaker

Superb file! Really great! The speed and flexibility is impressive. However, i'm experiencing some troubles. When I read this txt file:

Symbol,Date
XXH00,09/221999

it works great! But, when I read this txt file:

Symbol,Date
ESH00,09/221999

txt2mat cannot read any lines. I discovered, when the 2nd line starts with an 'E' or 'N', txt2mat somehow cannot match the format.

Is there a way around this or to fix this problem?

15 Nov 2011 Thomas

I wasted 3 hours searching for a solution to my problem before I stumbled upon this beauty! Thanks so much for all the effort you put into this. Good explanation and examples to get you started. Part of my permanent function collection..

14 Feb 2011 ESTEBAN

It works nice and smooth importing DEM (digital elevation models)...Thanks a lot

15 Dec 2010 Juan Pedro

Great job

10 Dec 2010 zampala ballarini

thanks a lot. Have been looking for this for a while.
great job

24 Nov 2010 Andres

Hi arouabm Ben Mohamed,
I'm sorry, txt2mat does not support mixed string/numeric output in cells as e.g. textscan does. It would be quite hard (probably) to change this without sacrificing speed and R13 compatibility.
Btw, you can use some features of txt2mat (like line filtering) with read mode 'char' and use textscan afterwards (see example 3b).
Perhaps, if ease of use is more important than speed, you may find the fex submission "readtext" useful as well.

19 Nov 2010 arouabm Ben Mohamed

Hi
thank you very much for this code it is great
but how can we make this function read a text file and convert it into a matrix containing words not characters
actually my file contains strings and numbers

14 Sep 2010 Matthew

Fabulously useful code - thanks Andres...

I am particularly impressed with the Param_Array t2mOpts and its' many flexible uses for reading complex data. I hope MATLAB considers standardizing your work.

Also BRAVO with your highly useful documentation!

13 Jul 2010 Yvan Lengwiler

I haven't looked at the code yet, but I have used this submission to read in 1 GB of data. It just worked beautifully, very fast, and allowed me to filter out bogus lines. Very useful submission.

14 Jun 2010 Akshay B

Thanks for your reply, Andres. Really, Mathworks should take your function and put it in their base functions - you have taken great efforts!!

16 May 2010 Andres

Thanks for your feedback. Indeed it would be nice to have a handy column selection input argument instead of using 'ConvString' with something like '%f %*f %f %*f %f'.
Regarding your memory issue, so far I can only refer to the 'NumericType' param and especially to example 6, but you probably found that in the doc already. txt2mat reads in the whole part of the file that is to be imported and initializes the output array. In an extreme case, a double array holding the data may consume about four times more memory than the file, unless some rows have missing values. I am thinking about making txt2mat more memory efficent - e.g. avoid slurping large files - without noticeably loosing speed (but tbh I have hardly any spare time now).

07 May 2010 Akshay B

An awesome and VERY helpful function!!! The param/value pairs are great - especially 'BadLineString' and 'RowRange'.

Small suggestion for future spins - if possible, having a 'ColumnRange" would be great.

Btw, I ran into an "Out of Memory" situation when processing a 355MB file, even though I have 3GB RAM and the 'memory' command reported that have atleast ~500MB for a single array (~1200MB for arrays). Any ideas how to fix this?

Thanks!!!

08 Apr 2010 Andres

It would be most simple if you could uniquely identify every line of text by some string (or some few strings), as with 'tex' (or just 'x') in your example:

txt2mat('example.txt','BadLineString',{'tex'})

If that is not possible, you could read in all lines and then remove all NaN-only rows:

A = txt2mat('example.txt',0,'ReadMode','line')
A = A(any(isfinite(A),2),:)

If things are more complex, you may also contact me via the 'Contact Author' link. Good luck!

08 Apr 2010 AwedBy Matlab

(I believe a 5 star rating is in order for this function anyway :-) )

08 Apr 2010 AwedBy Matlab

I tried using txt2mat to extract a matrix A from a file data.csv that has several lines of explicative text interspersed throughout the file, like this:

text 1
1 2 3
4 5 6
7 8 9
text 2
10 11 12

For the example above, txt2mat produced the matrix

10 11 12

rather than what I expected, which was

1 2 3
4 5 6
7 8 9
10 11 12

Can anyone help? Thanks!

18 Mar 2010 Ana Fernández García de Iturrospe  
04 Jan 2010 Pavan

Superb function. Worked without a problem. Great stuff.
Thanks Andres

23 Dec 2009 achus Pujante

Great function, you saved me a lot of time, thank you very much
Then I used your function to simplify my own case

16 Nov 2009 S. P.  
11 Nov 2009 Andres

@ Leonard
Thanks for pointing to the version number issue with the MCR (and that part of code you mention should have been replaced anyhow...). I've just sent you an email regarding a possible solution; if it got lost, please use the 'Contact Author' link on my author page.
Regards
Andres

09 Nov 2009 Leonard

txt2mat is excellent by being very straightforward in it's implementation. Could be the defacto standard within Matlab. Thanks Andreas for this.

If you are considering compiling a standalone application and deploying it using the MCR, you may want to consider the following:
I did not encounter any errors using this .m file while using it as long as MATLAB was installed on my machine (ver 7.5, BTW).
When deploying my executable to another machine and using the MCR (ver 7.7), the command line indicated the following:

Too many objects requested. Most likely cause is missing [ ] around left hand side that has a comma separated list expansion.

Error in ==> txt2mat at 519

515 %% Definitions
516
517 % find out matlab version as a decimal, up to the second dot:
518 v = ver('matlab');
519 vs= v.Version;
520 vsDotPos = [strfind(vs,'.'), Inf, Inf];
521 vn= str2double(vs(1:min(numel(vs),vsDotPos(2)-1)));

The .m file halted execution of my program because it was looking for a version # for MATLAB, which was not installed on the target machine. I patched the code by determining the matlab ver and editing the txt2mat.m as follows:

518 %v = ver('matlab');
519 vs = 7.5; %vs= v.Version;
520 vsDotPos = [strfind(vs,'.'), Inf, Inf];
521 vn= str2double(vs(1:min(numel(vs),vsDotPos(2)-1)));

Maybe it's possible to check for 'Matlab' on the target machine and/or the MCR and then handle this line appropriately.

Regards,
Len

05 Aug 2009 Val Schmidt

This seems like a terrifically useful tool.

One feature request. In addition to being able to specify characters that, if found in a line, mark the line for omission, it would be nice to be able to instead skip everything by default and specify characters of lines that are to be included.

Suppose, for example, you want to parse a log file. You'll want to extract all the log entries of a particular type. Rather than having to specify every other type of entry for omission, it'd be nice to be able to specify just the ones you want.

Thanks,
Val

22 May 2009 Gabriel Vézina

great code and fast to execute

24 Mar 2009 Bas

Needed to have a file import script that could handle a comma as the decimal separator. This worked instantly.

10 Feb 2009 Jose Miguel Jauregui

A great job,
It works perfectly when it comes to load very extensive text files, a very fast code. If you work with extensive text files, this is the code to use.

In case you need a modification Andres is always willing to provide help

Josemi

22 Jan 2009 Fredrik  
14 Jan 2009 Andres

@ John McArthur
The header itself *is* accessible as a string, e.g.

>> [A,ffn,nh,SR,hl] = txt2mat('myfile.txt');
>> headerNumbers = sscanf(hl,'%*s%f,%*s%f')

headerNumbers =
1001
400
40
15

(generally it is hard to guess which information in the header is useful to the user)

14 Jan 2009 John McArthur

Wondering if there's a way to get the program to recognize and import data in the header line. For instance, if you have the test conditions as header lines, like this:

nTimeInc, 1001, TotalTime, 400
ForcingFreq, 40, ForcingAmp, 15
T, X, Y, Z, Theta, Zeta
0, 0, 0, 0, 0, 1.0
0.4, 1.2, 0, 0, 0.01, 1.2
....

So, the header files have some useful info that would be nice to have accessible and can be added to plots and analysis.

Any thoughts?
johnnyfisma@hotmail.com

11 Nov 2008 ZY

A powerful code. I am using it (with some help from the author of the code) to read very complicated data files with headerlines throughout the file as well as data lines with different lengths (line folding).

And it is quite fast too.

Thanks Andres for your great work!

Zahra

03 Nov 2008 Ralf  
17 Oct 2008 djr djr  
27 Aug 2008 rodrigo abarca del rio

Excellent work. It helped me in reading data from different format, and skipping the headlines ... in just a second, after having tried over more than 4 hours by different methods. we are close to fortran now :-)
thanks so much a lot for your work.

02 Jun 2008 Andres T.

@wu zhiyong
Please see my posting in the newsgroup.

01 Jun 2008 wu zhiyong

Great work!
Thanks for your suggestion!
It helps me read the data from text files with ignoring the characters.But I fail to ignore the blank lines.
Do you have any suggestion?

23 May 2008 K AM

Fantastic script. Worked right out of the box. I used it bring in oddly shaped data files and it worked like a dream! Thanks Andres!

24 Apr 2008 Wladimir Alonso

It worked beautifully when I needed to read solar data tables (http://rredc.nrel.gov/solar/old_data/nsrdb/1991-2005/list_by_state.html) that contained headers, text and numbers (in this case using txt2mat(name_of_the_file,1,43,'ConvString',['%d-%d-%d,%d:%f' repmat(',%f',1,38)]) as kindly suggested by the author of the function)
thanks Andres!

29 Jan 2008 Florian H

Excellent! Finally a function that handles textfiles without the hassle of the builtin MATLAB ones.

I've combined it with
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=15294
to get a really usefull drag-and-drop import funtion for various textformats.

29 Jan 2008 vincenzo ficco  
Updates
08 Feb 2008

v05.61

03 Mar 2008

v05.62

07 Apr 2008

v05.86

20 May 2008

v05.86.1

21 May 2008

v05.90

27 May 2008

v05.96

26 Jun 2008

v05.97

03 Nov 2008

v06.01 · fixed bug: possible error message in file analysis when only header line number is given

02 Dec 2008

v06.04 · better handling of replacement strings containing line breaks (initiated by DS) · allow '*' wildcard in file name

09 Aug 2009

v06.12 · added line filter as requested by Val Schmidt

17 Nov 2009

v06.17.3 · new read modes 'char' and 'cell' to provide txt2mat's preprocessing without numerical conversion · enable 'good line' filtering during file analysis · better version detection, (hopefully) suitable for MCR (thanks to Len for his remark)

14 Dec 2012

v06.40 · better input argument check by inputparser, allowing input struct · minor changes in code and documentation

24 Mar 2014

v06.60 · added option to select lines by line number ('SelectLineFun'), e.g. to skip every n-th line as suggested by Kaare · reduced memory footprint and improved speed during good/bad line filtering by working in chunks

Contact us