Robust Data File Reading utility (RDFREAD)

Imports data from a comma- or tab-delimitted file.

RDFREAD handles comma-delimitted and tab-delimitted files and expects, but does not require the variable (column) names to be in the first line. Data may have missing values and character values mixed in the numeric data columns. All non-numerics in data lines are changed to NaN. This function does not properly read character variable columns, but instead turns such data into an entire column of missing values.

If the firstline has no numerics, it is assumed to be a header line with variable names. In this case a structure class is created with the fields name.varnames & name.data. If there is no headerline, a simple data matrix is created.

If there is no output argument (nargout=0), the results are put into base memory using filename as name.

examples
RDFREAD -- no input args, uses GUI file selection (uigetfiles)
RDFREAD(1) -- command line version
RDFREAD('file_name','C:\data_path\') -- works (path is optional)
xdat= RDFREAD('file_name') -- results placed in xdat

Below is an example of a messy data file that this utilty fucntion can read.

SECID,DATE,PRC,X1
10001,19970131,8.625,1
10002,1997215,13,
10003,,-15,4
1099a,o.0,.T,NaN
xd,,.,1.111
999,2,3,6

In this case, the RDFREAD 'data' results are

10001 19970131 8.625 1
10002 1997215 13 NaN
10003 NaN -15 4
NaN NaN NaN NaN
NaN NaN NaN 1.111
999 2 3 6

and filename.varnames=

'SECID' 'DATE' 'PRC' 'X1'

Note that this function is considerably slower than i/o routines
such as LOAD and DLMREAD because it reads and parses one line
at a time, but it works in a much more robust manner as long as you
do not need to read in 'character' columns.

10/25/2004 -- added block pre-allocation step to speed processing of large files