TDMS Reader

Read TDMS files v1 & v2 without DLL Current Version: 2.4

Description

Reads TDMS files into Matlab.

Advantages:
   - supports reading v2 files
   - doesn't require the NI DLL, thus it doesn't require 32bit windows
   - supports interleaved data
   - allows only reading names & properties to get a quick feel for what is in the file
   - allows reading specific subsets of the data for limited memory usage

• MATLAB release: MATLAB 7.10 (2010a)

[Joacim]

Impressive amount of work you've put into this. Good job and well documented.
Tested the function on my data saved by Signal Express 2009. Works for small files (<6 MB) but when trying to load larger files (>50 MB) it crashes.
I'm lazy so I downloaded the matlab_tdm_example_sp2010.zip from NI. Using the nilibddc.dll and approx. 200 lines of code, including comments, I'm reading even larger files (approx 320 MB) in less than 4 seconds.

Below is some info from the crash. Maybe it can help you fix the problem:

testTDMSreading
266 propName = fread(fid,propNameLength,'*char')';
??? Error using ==> fread
Out of memory. Type HELP MEMORY for your options.

Error in ==> TDMS_preprocessFile at 266
                propName = fread(fid,propNameLength,'*char')';

Error in ==> TDMS_readTDMSFile at 176
[rawDataInfo,numberDataPoints,objectPaths,segInfo] =
TDMS_preprocessFile(fid2,isIndexFID2,params);

Error in ==> testTDMSreading at 34
output = TDMS_readTDMSFile(fullfileName,optionCA{:});
 
>> testTDMSreading
266 propName = fread(fid,propNameLength,'*char')';
K>> propNameLength
propNameLength =
  1.9629e+009
K>> fid
fid =
    12
K>> curProps
curProps =
  Columns 1 through 8
    'format-string' [1x22 char] 'unit_string' [] [] [] [] []
  Columns 9 through 20
    [] [] [] [] [] [] [] [] [] [] [] []
K>> iProp
iProp =
     4
K>> propNameLength
propNameLength =
  1.9629e+009

[Jim Hokanson]

Hi all,

  In response to Joacim's comments:

1) There should be no file size limitation (other than the users memory size), the out of memory indicates a parsing error.

2) I am still quite unsure as to how well the code handles unicode. The error seems to indicate an ability to properly parse a string ...

3) If anyone sees a similar error please feel free to email an example file to me and I'll gladly try and fix the code so that it is handled.

Thanks,
Jim

[Jim Hokanson]

Joacim,

  I fixed the unicode bug. Ver. 1.1 should be up soon. I had misread some documentation regarding # of bytes vs # of characters. Thanks for pointing out my mistake.

Jim

[Ray]

 

[Edward Zechmann]

 

[Harm]

 

[Harm]

Works great! The NI tool with the DLLs failed on me (64bit version does not seem to work, 32 bit version was OK).
But this one is better; and I think it will work on Linux systems as well.

[Guosong Zhang]

??? Error: File: TDMS_readTDMSFile.m Line: 453 Column: 11
Expression or statement is incorrect--possibly unbalanced (, {, or [.

[Jim Hokanson]

@Guosong Zhang
In 2009b the ~ was introduced as a way to ignore outputs. This can be especially useful if that output requires a lot of memory. I've gotten in the habit of putting those in although I realize not everyone may be using 2009b or later. I might upload a fix in which I put dummy variables back, although I'd be tempted instead to write a small script that allows checking the version and doing a replace in all the files if needed.

[Kevin]

please advise if this works with 64bit matlab

[Jim Hokanson]

Yes it does. I use it with 64 bit Windows and it has been tested with a 64 bit Mac.

[Fiona]

Thanks for uploading!
When I tried to read my TDMS file, I got the errors below. I wonder if anyone can help. Thanks!

??? Error using ==> TDMS_preprocessFile at 375
The remaining data doesn't split evently into chunks, estimated # of chunks: 5.000000e-001

Error in ==> TDMS_readTDMSFile at 198
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> Test at 11
[finalOutput,metaStruct] = TDMS_readTDMSFile(tdmsFileName)

 

[Jim Hokanson]

Hi Fiona,

   Thanks for pointing out the issue. I'm really busy early this week but I might be able to look at this issue Thursday or Friday. See me email for more details.

Jim

[Juha Suomalainen]

Hi,
Great code you have here but I think there is still a bug somewhere inside it. I am able to load my whole TDMS file using you function. That works fine. Unfortunately some of my datasets will be huge and due to memory issues must be loaded and processed in parts. If I try to read a subset of data using your function, I get an error.

Here is my code to read and process data group by group:
---
% Get list what is inside the TDMS
params = {'GET_DATA_OPTION','getnone'};
TDMS_Structure = TDMS_readTDMSFile(TDMS_File, params{:});
% Read and process groups from the TDMS file one by one
for GroupNumber = 1:numel(TDMS_Structure.groupNames)
    params = {...
        'GET_DATA_OPTION', 'getSubset',...
        'OBJECTS_GET', struct('groupsKeep',{{TDMS_Structure.groupNames{GroupNumber}}})};
    TempData = TDMS_readTDMSFile(TDMS_File, params{:});
    % process subset of data here
    ProcessData(TempData)
end
---

I can read the TDMS_Structure without problems, but the subset loading call of 'TDMS_readTDMSFile' produces an error:

---
??? Error using ==> TDMS_readFileHelper_v1 at 211
The # of requested values does not equal the # of returned values, error in code likely

Error in ==> TDMS_readTDMSFile at 229
    data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);
---

Is it a bug or am I just using your code with wrong parameters?

[Jim Hokanson]

Hmmm,

   In general the code looks fine. I would throw in the already parsed meta structure back into the function call to save processing time. I'll send some instructions on that later. As the code states when a problem like that happens it is most likely my fault. In general I try and throw in checks where possible because I'd rather have things not work and tell you than pretend to work. I'll send you an email with more instructions on how we can find a fix to this problem.

Jim

[Jim Hokanson]

Found bug for v2.2 effecting use of GET_DATA_OPTION with values 'getSubset' or 'ignoreSubset'
This bug will already throw an error to let you know the read was incorrect

If you get the error:
The # of requested values does not equal the # of returned values, error in code likely
Then this should be fixed by adding this line:
numValuesToGetActual(~keepDataArray) = 0;

after these lines: (around line 133)
useSubset = false;
subsetInfo = struct([]);
numValuesToGetActual = metaStruct.numberDataPoints;

This will be fixed in the next version.

[Craig Smith]

Jim, I've just downloaded your files and tried to read a TDMS file v2 astarting with what you have in the readMe.txt file. When I run TDMS_getStruct(filename) and select my file I get the following error messages:
??? Undefined function or method 'TDMS_handleGetDataOption' for input arguments of type 'struct'.

Error in ==> TDMS_readTDMSFile at 192
TDMS_handleGetDataOption('check',paramsStruct)

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Any ideas how I should proceed?

[Craig Smith]

Jim, Please ignore my earlier comment. I have resolved (stupid user error by me). I now get to the following error:
 my_tdms_struct = TDMS_getStruct(filename);
??? In an assignment A(I) = B, the number of elements in B and
 I must be the same.

Error in ==> TDMS_readTDMSFile at 255
        groupIndices(curGroupCount) = I_groupObject;

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

The value of curGroupCount is 1 but the I_groupObject is an array of length 14 with integer values of 2 through 15. Not sure why the mismatch here

[Jim Hokanson]

Craig has identified a 2nd bug that is present in v2.2. This bug effects all previous versions. Group names with "/" characters are currently incorrectly parsed. An update to v2.2 will be released soon that addresses this bug, and the getSubset,IgnoreSubset bug. NOTE: Both of these bugs will cause errors instead of failing silently. Thanks guys for letting me know you had a problem so I could improve the code.

Jim

[Jeffrey Fletcher]

Great piece of work - thanks.
I get an erro when trying to read by TDMS file created directly by the NI-DAQmx .NET TdmsLogging function. The error stack is:

??? Error using ==> TDMS_processLeadIn at 50
Currently code is unable to ignore/handle Raw Daq MX data
Error in ==> TDMS_preprocessFile at 175
    [flags,info] = TDMS_processLeadIn(fid,lastLetter);
Error in ==> TDMS_readTDMSFile at 225
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);
Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});
 
It appears that "Raw DAQmx data" is not currently supported. It's too bad because you can create a data logger in MATLAB (with ONLY NI-DAQmx and base MATLAB installed) with a few lines of code. I'll share this with anyone interested.

Any chance of an update that supports this data type?
Jeff

[Jim Hokanson]

That shouldn't be a problem. The raw format is currently unsupported because its type is unspecified.

[Andreas]

Hi Jim,

It looks like your script does exactly what I need. But when I tried to use it today, I get the following error with my tdms file:

>> test=TDMS_readTDMSFile('test.tdms')

??? Undefined function or variable "dataType".

Error in ==> TDMS_readFileHelper_v1 at 106
                    temp = fread(fid,numValuesRead*nChans,precisionType{dataType});

Error in ==> TDMS_readTDMSFile at 258
    data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);
 
Reading the index file is working fine and i get the following result:

 >>test=TDMS_readTDMSFile('test.tdms_index')

test =
              rootIndex: 1
           groupIndices: 2
             groupNames: {'Analog Input'}
            chanIndices: {[3 4 5 6 7]}
              chanNames: {{1x5 cell}}
                   data: {[] [] [] [] [] [] []}
              propNames: {{1x1 cell} {1x0 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}}
             propValues: {{1x1 cell} {1x0 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}}
        objectPathsOrig: {'/' '/'Analog Input'' '/'Analog Input'/'T'' '/'Analog Input'/'U'' '/'Analog Input'/'I'' '/'Analog Input'/'B'' '/'Analog Input'/'H''}
    numberDataPointsRaw: [0 0 28000 28000 28000 28000 28000]
               dataType: [0 0 10 10 10 10 10]
           dataTypeName: {'void' 'void' 'double' 'double' 'double' 'double' 'double'}

Maybe you have a tip for me what I have to change in the script to read my tdms file?

Thank you in advance.

Best regards

Andreas

[Jim Hokanson]

Hi Andreas,

  I've known about that bug for a while I just haven't uploaded a fix yet. I had been a bit slow since I wanted to incorporate some other changes and it is an error that will be thrown as opposed to failing silently and pretending to work. I'll upload a modification very soon but for now you just need to replace in that particular case structure (starting around line 100) in TDMS_readFileHelper_v1 the following:
dataType -> change to -> dataTypes(1)

Jim

[Andreas]

Does the Code work with Matlab 7.11 ?

I've tried it on a 32bit (XP) and a 64bit (Windows7) System with various TDM/TDMS files.
Depending on the type of input file I get 2 different error messages. For native sample files from NI the reaction is:

>> my_tdms_struct = TDMS_getStruct(filePath);
??? Error using ==> TDMS_processLeadIn at 33
Unexpected lead in header

Error in ==> TDMS_preprocessFile at 179
    [flags,info] = TDMS_processLeadIn(fid,lastLetter);

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});
 

My self-generated TDMS files (read with NI Diadem and Excel importer without problems) gave

??? Subscripted assignment dimension mismatch.

Error in ==> TDMS_preprocessFile at 456
            rawDataInfo(curIndex).dataMatrix(indices,2) = ...

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Is it a Matlab version issue or am I missing something else?

Best regards,
Andreas
[I'm a different Andreas than the one who posted here before..]

[Jim Hokanson]

Andreas, I am unaware of any bugs in the code at this time. Please check the message I sent you.

[Blaine]

Is it possible to only read sections of a large (~3.5 GB) TDMS file with your code? I would like to work with pieces of the data at a time. If so, how would I do it?

[Jim Hokanson]

You can be as specific about reading data as you would like. See the documentation file TDMS_retrievingSubsets. You can get as fine grained in reading of the data as you would like, specifying down to the channel and sample numbers if you like. Please let me know if there is anything about the documentation you think should be clarified.

Looking over this more carefully it appears that I have made an error in my documentation related to the meta_struct. I will be updating that soon.

The main error is that subsequent calls is that if you read more more data and want to use the metaStruct from a previous call to reduce parsing time, you still need to pass in the filename again:

TDMS_readTDMSfile(filename,'META_STRUCT',metaStruct,etc)

I tend to recommend using TDMS_getStruct, in which case you would use:

TDMS_getStruct(filename,versionNumber,{'META_STRUCT' metaStruct etc})

[Blaine]

I tried to open my large data file with:

[tmp,metaStruct] = TDMS_readTDMSfile('C:\Users\bjmarti4.NDC\Desktop\FIRST_LIGHT_CO2.tdms','GET_DATA_OPTION','getnone')

I get the following error:

??? Error using ==> TDMS_processLeadIn at 50
Currently code is unable to ignore/handle Raw Daq MX data

Error in ==> TDMS_preprocessFile at 179
    [flags,info] = TDMS_processLeadIn(fid,lastLetter);

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_exampleFunctionCalls at 95
[tmp,metaStruct] =
TDMS_readTDMSfile('C:\Users\bjmarti4.NDC\Desktop\FIRST_LIGHT_CO2.tdms','GET_DATA_OPTION','getnone')

I was able to open a small test file, but when I try to open my data file is when I get the error

[Jim Hokanson]

Regarding improvements in documentation, that is probably one area to improve. Raw DAQ MX data is a proprietary Labview format and I've been turned down in requests to get specifications on that type. I've managed to parse a couple of examples but since I have had so few examples to go off of I haven't tried implementing it. In my experience the raw format is really just read as a normal data type (int16 in my experience??) with properties that specify how to scale that raw data into measured data. I don't know how to extract the real data type from their meta information. I could technically just treat it as some data type, say uint8, but to parse the data generally (not always) relies on knowing the size of the data type. In addition there is some strange buffering of the meta information which I can manually hard code around but I am not sure if it is always present.

That being said, if you have an example file I would be interested in looking at it so that I could eventually offer some workarounds. As I said I have had some luck in extracting the data.

[Blaine]

I'm new to TDMS files I just assumed that they can all be read the same. The test file I have is just an example file I downloaded from NI. I wasn't aware that Raw DAQ MX was different than any other TDMS file.

[John Vazey]

I get an error
??? Error using ==> TDMS_readFileHelper_v1 at 188
Catastrophic error detected, code probably has an error somewhere

Has anyone else had this problem?

JV

[Sergiy]

 

[Laura]

It is some time that I am playing with your routines. It is a fantastic work. However, although it is working fine to convert to matlab a complex .tdms file, I didn't succeed in modifying the precision of the converted values. I guess is specified in the readOptions (different from default), and specifically called in the subroutines TDMS_getTypeSeekSizes and TDMS_readFileHelper_v1. I am not a advanced matlab user. I would really appreciate if you could help me to change those settings.
My problem consists in that I would like to have the record the structured data in double precision instead of in single.
Thanks for your work and attention

LV

[Jim Hokanson]

Laura, you are pointing out an issue that I thought would be good to address but that I just haven't gotten to yet. In general I have found it is good to respect the data type so as to prevent memory overflow for a user, but there are obviously times when you want things read in as a double.

I eventually wanted to handle this more gracefully but you can try the following hack to get things to work:
1) in TDMS_initData, remove the string specification from zeros, in your case it sounds like you want case {9 25} to be:
output = zeros(1,nSamples) instead of output = zeros(1,nSamples,'single')
2) in TDMS_getTypeSeekSizes change the read specification to be "type" instead of "type=>type", in your case you want 'single' instead of 'single=>single'

These two changes should make it so that all doubles are read as singles. I eventually want to make a cleaner fix but I've been too busy lately with other things.

[Laura]

 

[Laura]

It worked perfectly!Thanks a lot for the program and for the fast help!
It is really the best work I have found online for tdms conversion

[Michael Kung]

 

[Dedric Xu]

Hi Jim,
I've encounteered a maddening problem. I have tried to use your function to read my tdms file,but failed. And my file is a little larger, about 60MB.

When I ran your function, it crashed.

The function I used is [output,metaStruct] = TDMS_getStruct(filePath,4);

And the program has ran about 20min,then it crashed.

I gtt the following error:

Warning: Warning: FOR loop index is too large. Truncating to 2147483647.
> In TDMS_readFileHelper_v1 at 66
  In TDMS_readTDMSFile at 249
  In TDMS_getStruct at 57
Error using TDMS_readFileHelper_v1 (line 184)
Catastrophic error detected, code probably has an error somewhere

Error in TDMS_readTDMSFile (line 249)
    data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Is it a Matlab version issue or am I missing something else?

Best regards,
Dedric

[Jim Hokanson]

Dedric, please check my email. Thanks.
Jim

[Brian Kaul]