Code covered by the BSD License  

Highlights from
TDMS Reader

4.77778

4.8 | 23 ratings Rate this file 171 Downloads (last 30 days) File Size: 47.7 KB File ID: #30023

TDMS Reader

by

 

13 Jan 2011 (Updated )

Read TDMS files v1 & v2 without DLL Current Version: 2.5

| Watch this File

File Information
Description

Reads TDMS files into Matlab.

Advantages:
   - supports reading v2 files
   - doesn't require the NI DLL, thus it doesn't require 32bit windows
   - supports interleaved data
   - allows only reading names & properties to get a quick feel for what is in the file
   - allows reading specific subsets of the data for limited memory usage

Required Products MATLAB
MATLAB release MATLAB 7.10 (R2010a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (99)
23 Jun 2014 Stefan

Hi Jim,
I also have the 'Unexpected lead in header' error but when I try the fix you mention:
'This problem can be fixed by replacing the error you are seeing with:
flags = [];
info = [];
eof_error = true;
return;'

I get an error:
WARNING: File was not closed properly.
Data will most likely be missing at the end of the file
In an assignment A(:) = B, the number of elements in A and B must be the same.

Error in TDMS_readFileHelper_v1 (line 148)
data{I_object}(startI:endI) =
fread(fid,numValuesAvailable,precisionType{dataType});

Error in TDMS_readTDMSFile (line 252)
data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Any idea?
Stefan

03 Jun 2014 Jim Hokanson

@Benoît

I have no idea what would be causing the problem you are referencing. Can you send me an example file that has this problem? I'll send a private message with more details.

Jim

03 Jun 2014 Benoit

Hi Jim,

Thanks for your job.
I tried to use your code, and I don't have any error, but the file I get is empty. My original file is 70Mo, and the final one is only 10Mo.
Do you have any clue to help me ?
Thanks a lot.

Benoît

14 Feb 2014 Jim Hokanson

@David

I've really wanted to change that interface for a while :/

The issue is that the input should be a structure ARRAY that contains strings.

getStruct.group = '6120';
getStruct.channel = 'Untitled 3';

Feel free to email if you need more help.

Jim

13 Feb 2014 David DeVilbiss

Jim,

back on 22-Feb-2014, Joe Cuschieri had a question concerning reading in segments of data using your functions. I'm getting the same errors. Did you have a solution for him or some sample code to simple read in segments of data.

Thanks, Dave

04 Feb 2014 Jonathan Currie

Great bit of code, worked first time, just how it should be!

21 Jan 2014 Jim Hokanson

@Joe,

National instruments won't release the specs for their daqmx format (I've asked), so I don't know what their binary data means. They recently updated their dll to support reading version 2 files, so you might have luck using that. I've been meaning to look at the dll but it's really low on my priority list at this point.

The link to the dll can be found at this page:
http://digital.ni.com/public.nsf/allkb/A3663DE39D6A2C5A86257204005C11CA

There used to be a specific page for the dll (in which it said it supported only version 1, and then recently version 2) but I can't find it anymore.

Good luck and feel free to email me if you have any more questions.

Jim

21 Jan 2014 Joe

Hello,

I get a strange error when trying to use TDMS_readTDMSFile:

It tells me:
"Error using TDMS_processLeadIn (line 50)
Currently code is unable to ignore/handle Raw Daq MX data

Error in TDMS_preprocessFile (line 179)
[flags,info,eof_error] = TDMS_processLeadIn(fid,lastLetter);

Error in TDMS_readTDMSFile (line 226)
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);"

Surely the program is fit to handle raw DAQmx data right? How can I avoid this error?

22 Oct 2013 Jim Hokanson

@Bob,

Sorry about the obtuse error message. In the new version I've been working on I am hoping to provide links to lengthy documentation on what each error message actually means. Each section of the tdms file starts with the characters tdsm (data file) or tdsh (if it is the index file). If those are not present, it indicates that there was either an error in parsing the file, or in writing the file. I think every time I've seen this error, the issue has been that the file is not closed properly (using tdms close in Labview) and the last segment is corrupted.

This problem can be fixed by replacing the error you are seeing with:
flags = [];
info = [];
eof_error = true;
return;

If you have any more problems feel free to contact me via email.

Jim

21 Oct 2013 Bob

I'm trying to help a coworker decode a message he is getting reading tdms files I'm creating. We read a lot of files and we occasionally see an error with the following format:

Error getting TDMS struct: 2013-10-17 16_50_15_902.tdms
MException with properties:

identifier: ''
message: 'Unexpected lead in header'
cause: {0x1 cell}
stack: [6x1 struct]

Is there anything that explains what this error means so I can correct it?

Thanks!

17 Oct 2013 Jim Hokanson

@Kadir,

Please get in touch with me via email and perhaps send a complaint to NI.

Jim

17 Oct 2013 Kadir

@ Jim,

Thanks a lot for the clarification. I don't have Excel. I tried the dll but it takes only first 1000 values of the sample. Therefore, I'd like to go for the 3rd option.

What do you mean by "regular array" and how can I perform this conversion?

Thanks again, Kadir

16 Oct 2013 Jim Hokanson

@Kadir,

That flag is updated on every segment, and is not constant throughout the file. The format of that type of data is not specified by NI even after I requested it. Alternatives include:
- trying the Excel TDMS reader
- trying their new dll code which might support Raw DaqMX code
- converting the data in Labview from Raw DaqMx into a regular array, and then using my code

Feel free to email me if you have any more questions.

Jim

16 Oct 2013 Kadir

Hi,

I got the error "Currently code is unable to ignore/handle Raw Daq MX data". Then, I looked at TDMS_processLeadIn, line 49-51:

if flags.hasRawDaqMX
error('Currently code is unable to ignore/handle Raw Daq MX data')
end

Then, I checked the value of "hasRawDaqMX" field of the struct "flags". It was 0. Then, I shouldn't get the error which is implemented in the line 50, right?

Does anybody have any idea what's going on?

Thanks in advance, Kadir

11 Oct 2013 Harry

@Jim

Works perfect for me! Great!

Thank u very very much!!!!!

10 Oct 2013 Jim Hokanson

@Harry,

You need to add the tdmsSubfunctions folder to your path as well. I had originally placed those in a separate folder to try and make it more obvious which functions you would call and which ones you wouldn't. Sorry about the confusion.

Jim

10 Oct 2013 Harry

Hey Jim,

I got a problem with your code.
If I want to convert a .tdms file, I always receive errors.

for example:
>> filename = 'C:\Users\*****\Desktop\v2p5\tdms\Messdaten.tdms';
>> my_tdms_struct = TDMS_getStruct(filename);
Undefined function 'TDMS_handleGetDataOption' for input arguments of type 'struct'.

Error in TDMS_readTDMSFile (line 216)
TDMS_handleGetDataOption('check',paramsStruct)

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Can u explain, whats the problem?

07 Oct 2013 Jim Hokanson

Hi all,

I recently came across a bug in my code which I expect impacts very few people and for the majority of people it impacts, the following error will most likely be thrown "The remaining data doesn't split evently into chunks." That being said, it is a bug and I'll try and get a fix out soon. From what I can tell it would cause a problem if you occasionally decide not to write a subset of a group of channels, and at other times you write the entire group. Something like:

If case1
channels = 1:3
else
channels = [1 3]
end

In my opinion this error actually goes against the NI documentation so I need to contact them to for clarification.

More details to follow.

Many thanks to Junghwan Kim for sending a sample file to me which allowed me to finally understand this problem.

Jim

07 Oct 2013 Henrik

Hi!

I am getting the following error while running step 2,3 and 4 in your example file "TDMS_exampleFunctionCalls".
Can you please help me?

Error using fseek
Invalid file identifier. Use fopen to generate a valid file identifier.

Error in TDMS_preprocessFile (line 155)
fseek(fid,0,1);

Error in TDMS_readTDMSFile (line 226)
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Best Regards
Henrik

16 Sep 2013 Jim Hokanson

@Navid,

Given that NI won't release the data format specs for RawDaqMX data, your options are the following:
1) Tell NI you want the specs released
2) see 1)
3) Try this:
http://www.ni.com/white-paper/4906/en

Good luck

16 Sep 2013 Navid

Dear Jim,
Thanks for your fast answer,
you were right about wrong code but after using your code I got same errors as Jury and Jane,unfortunately I am not able to resave the data,is there other ways to get around this problem?!

many thanks..

15 Sep 2013 Jim Hokanson

@Navid,

It seems like you might be using different code:
http://www.mathworks.com/matlabcentral/fileexchange/28771-converttdms-v9

I would recommend you using my code instead. Let me know if you still have problems.

Jim

15 Sep 2013 Navid

Dear Jim,
at first thank you so much for your code,I have problem with some kinds of TDMS files,could you help me plz?! I get this:
Converting 'ADC_20130808_165723.tdms'...Undefined function or variable 'TDMSFileNameShort'.

Error in convertTDMS>getSegInfo (line 405)
e=errordlg(sprintf(['Seqment %.0f within ''%s'' has interleaved data which is not
supported with this '...

Error in convertTDMS (line 276)
[SegInfo,NumOfSeg]=getSegInfo(fid);

27 Aug 2013 Junghwan

Hi Jim,

Firstly, I deeply appreciate for your code.

I get the same error as Robert and Fiona.
It looks like you answered Fiona via email. Could you share your answer to Fiona?

Many thanks!

Error using TDMS_preprocessFile (line 450)
The remaining data doesn't split evently into chunks, estimated #
of chunks: 5.000000e-01

Error in TDMS_readTDMSFile (line 226)
metaStruct =
TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

06 Aug 2013 CLF  
04 Aug 2013 Jim Hokanson

@Jane

The short answer is that there is no solution. The format of Raw Daq MX data is proprietary and NI has refused my request for documentation.

The easiest way to get around this if you are still looking to use my code is to read the data file using Labview and to resave the data not using Raw Daq MX.

04 Aug 2013 Jane

Hello!

I have been having the same error as Jury. Could you possibly post a solution?

Cheers!

13 Jul 2013 Jim Hokanson

@Jury

Please send me an email regarding this problem.

12 Jul 2013 Jury

Hi!

I try to read TDMS file, created LV2012 (12.0.1f2) in Matlab R2013a, 64bit, and obtain error:

Error using TDMS_processLeadIn (line 50)
Currently code is unable to ignore/handle Raw Daq MX data

Error in TDMS_preprocessFile (line 179)
[flags,info,eof_error] = TDMS_processLeadIn(fid,lastLetter);

Error in TDMS_readTDMSFile (line 226)
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Is any suggestion for reading TDMS file?

Jury

21 Jun 2013 Jim Hokanson

@Cole ??? - What fix? Please contact me via email to discuss specifics.

Thanks,
Jim

21 Jun 2013 Cole Van Vlack

I should also mention I did try the fix you gave before. When I use it, I get the following error,

---------
Error in TDMS_processLeadIn (line 27)
Ttag = fread(fid,1,'uint8');

Output argument "eof_error" (and maybe others) not assigned during call to "E:\Data\Cobra Integration
Test\v2p5\tdmsSubfunctions\TDMS_processLeadIn.m>TDMS_processLeadIn".

Error in TDMS_preprocessFile (line 179)
[flags,info,eof_error] = TDMS_processLeadIn(fid,lastLetter);

Error in TDMS_readTDMSFile (line 226)
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Error in test (line 4)
my_tdms_struct = TDMS_getStruct(filename);

08 Jun 2013 Zeb

Worked fine on Matlab 2010, now trying to use with Matlab 2008.

Getting this error:
"??? Error: File: TDMS_readTDMSFile.m Line: 134
Column: 3
Expression or statement is incorrect--possibly
unbalanced (, {, or [."

How do I solve this?

04 Mar 2013 Jim Hokanson

Amrita,

Please add the tdmsSubfunctions directory to your path as well.

Jim

04 Mar 2013 Amrita

I get the following error when I try to read a tdms file:

??? Undefined function or method 'TDMS_handleGetDataOption' for input
arguments of type 'struct'.

Error in ==> TDMS_readTDMSFile at 216
TDMS_handleGetDataOption('check',paramsStruct)

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

How do I solve this?
Thanks!

01 Mar 2013 Robert

I am trying to use the file to convert tdms file but I have a following error:

Error using TDMS_preprocessFile (line 450)
The remaining data doesn't split evently into chunks, estimated # of chunks: 8.181818e-01

Error in TDMS_readTDMSFile (line 226)
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Could someone help me to solve it, please?
Thanks

22 Feb 2013 Joe Cuschieri

I am using your algorithm posted on the MATLAB Central to read data from a TDMS file, however I am getting an error which I do not quite understand why.

This is a section of my code:
getStruct = struct;
getStruct.group = {'6120'};
getStruct.channel = {'Untitled 3'};
getStruct.indices = [2 3];
tempOutput4 = TDMS_readTDMSFile(fname,'GET_INDICES',getStruct,'SUBSET_IS_LENGTH',false)

However I get these errors:
Error using cell/ismember>cellismemberlegacy (line 132)
Input A of class cell and input B of class cell must be cell arrays of strings,
unless one is a string.

Error in cell/ismember (line 76)
[varargout{1:nlhs}] = cellismemberlegacy(varargin{:});

Error in TDMS_handleGetDataOption (line 167)
[~,loc] = ismember(objPaths_getIndex,objectPaths);

Error in TDMS_readTDMSFile (line 228)
optionStruct = TDMS_handleGetDataOption('getArray',paramsStruct,metaStruct); %Option
handling

Error in Test1 (line 21)
tempOutput4 =
TDMS_readTDMSFile(fname,'GET_INDICES',getStruct,'SUBSET_IS_LENGTH',false)

I am using MATLAB R2012b.

Any suggestions?

22 Feb 2013 Jim Hokanson

René,

TDMS doesn't contain time unless you explicitly save it as a channel. This is most often only done if you are sampling something periodically, such that the timing between events is not constant. Otherwise time is usually implied from a known sampling rate. If you are saving the time as a timestamp, I currently return the timestamp array as a numeric vector in Matlab's time format instead of converting each element to a string (like datestr). This makes it easier to do computations on the time (note that a difference of 1 in this vector represents a difference of 1 day, divide by 86400 to go to seconds). If you still have questions feel free to contact me via email.

Jim

21 Feb 2013 Rene

Jim,

Like your work a lot. It will save a lot of disk space, since it will be no longer required to save as lvm.

The output format is pretty straightforward, but I however can't figure out what variable contains the array with sample times.
I figure the tdms-file must hold the time since excel can grab it from there.

Could you tell me whether the sample time is also imported into matlab?

Best regards,
René

08 Jan 2013 Jim Hokanson

Hi Anthony,

Yes this problem is solved. I tend to send emails offline.

This is the email I sent to Kevin. Let me know if you need anything else:

EMAIL TO KEVIN:
Whoever is giving you the data is most likely not closing the files properly. In addition they are writing the files really inefficiently.

In general every time you write to the file a additional index header is created. It is more efficient to write 100k values to a file than to write 1 value, 100k times with 100k headers. In this case the NAV info is being written in one sample, then the TANDEM is being written for 200 samples. It would be more efficient to write these to some queue in memory and then dump them to file occasionally.

That being said you can read a portion of your files if you change the following:
Assuming you are not doing any fancy reading:
- TDMS_readFileHelper_v1 - comment out near line 187 which says if eofPosition ~= ftell(fid)
- In TDMS_processLeadIn - around line 34, comment out the error and add the following:
flags = [];
info = [];
return
- In TDMS_preprocessFile - around line 180, write the following:
if isempty(flags)
nSegs = nSegs - 1;
break
end

Essentially you are just stopping the code when you get to the error and returning only what you have up until that point. What you have in that file is alot already. My guess is you are only missing a bit of data at the end but I haven't bothered to check that ...

A few others have had similar errors but there is a flag in the header which is specifically for ending too early. This is the first time I have seen the header itself be corrupt like this. Perhaps that is due to working with the tdms file and not the index file.

Good luck and let me know if those changes get things working for you or not.

08 Jan 2013 Anthony

Jim,

Was there ever a solution to Kevin's error? I seem to be getting the same error when i run the Matlab code.

08 Jan 2013 Harm

@uwe: I have used this function on 64bit Matlab (linux and win64) without any problem.

08 Jan 2013 Uwe Lelke

Can I also read simple TDM-files with 64-bit Matlab?

09 Oct 2012 Jim Hokanson

Farhan: Sorry about the delayed response. I'm getting ready for a conference and I'm in a bit of a rush to finish things up. I'm glad you got things working. I'll add a check for your error so that others get a more obvious error message. Thanks!

Kevin: File size shouldn't be an issue. In fact I wrote the code specifically to support very large files in which you can do partial reads and process the data before reading more. See my private message on how we might be able to resolve your issue.

09 Oct 2012 farhan

Jim,
I resolved the issue, there were some empty Group names that our LabView code was forgetting to assign. We fixed that problem in the LabView code and now the TDMS converts beautifully into a data structure.
Thank you!

09 Oct 2012 Kevin

Many thanks for this work it's great but I have an issue when I try to read a bigger file : few MB works well but with a 80 MB file I get this error :
"
??? Error using ==> TDMS_processLeadIn at 33
Unexpected lead in header

Error in ==> TDMS_preprocessFile at 179
[flags,info,eof_error] = TDMS_processLeadIn(fid,lastLetter);

Error in ==> TDMS_readTDMSFile at 226
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});
"
Anyway it's working great for the smaller files.

08 Oct 2012 farhan

Jim,
I successfully used your code with an older version of our TDMS, but now I am getting an error. The file is intact in that it can open in the Excel reader.

>> TDMS_readTDMSFile('LOG_2012-10-08_10-53-27.tdms')
??? Improper assignment with rectangular empty matrix.

Error in ==> TDMS_readTDMSFile at 275
groupIndices(curGroupCount) = I_groupObject;

Any help would be greatly appreciated!

17 Sep 2012 Pian Xiaochuan

Jim, I really appreciate your work. It helps me a loooooooooooooooooooot!!!

Marshall

09 Aug 2012 Larry

This code was a god send for me. Very well written and documented, functional and efficient. Thanks so much for contributing this to the public, Jim.

28 Jul 2012 Jim Hokanson

Dedric Xu recently reported observing a problem with the code which turned out to be a bug. If I really think hard, I convince myself that it might be possible to have the bug he was observing occur without any noticeable code failure. I would recommend updating the code to V2.5 when you can (once it posts). The bug came from interpreting NI documentation too literally. They have acknowledged the poorly worded documentation and said they will update it. My apologies to anyone else this effected, although I sincerely expect that it was basically no one.

More details can be found in the version info file.

Although I am currently working on other things, I still have in mind the following updates:

1) Port of the code to an object oriented nature. Part of this involves creating more functions. I stayed away from this due to speed concerns with function call overheads and data passing but my decision obviously makes debugging and updating more difficult, as well as potential "optional" mex substitutions at key bottleneck points.
2) A TDMS viewer GUI
3) Support to optionally read data directly into a double vector, instead of the format it was written in and converting after the fact.

25 Jul 2012 Toby scientis

After lots of frustration with the code that NL wrote, this worked perfectly on the first try.

13 Jul 2012 Jim Hokanson

You need to add the subfunction folder to your path as well. I eventually want to make this a class which makes some methods private and hidden, but in the mean time the sub functions folder just helps delineate between functions you should call and functions you shouldn't.

13 Jul 2012 Hari Patel

I tried to read .tdms file with what you have in readMe.txt and I got following error:
>> filename = 'Digital_Input.tdms';
>> my_tdms_struct = TDMS_getStruct(filename)
data = TDMS_readTDMSFile ('Digital_input.tdms')

??? Undefined function or method 'TDMS_handleGetDataOption' for input arguments of type
'struct'.

Error in ==> TDMS_readTDMSFile at 213
TDMS_handleGetDataOption('check',paramsStruct)

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

??? Undefined function or method 'TDMS_handleGetDataOption' for input arguments of type
'struct'.

Error in ==> TDMS_readTDMSFile at 213
TDMS_handleGetDataOption('check',paramsStruct)

28 Jun 2012 Laura  
28 Jun 2012 Laura

Thanks a lot for the reply and indeed also again the solution Jim, working perfectly!!
(I didn't realized I had done such a change, my mistake)
Thanks again

26 Jun 2012 Jim Hokanson

John:

Thanks for the helpful comments. I started writing this file in late 2010 since v2 TDMS files weren't supported by the NI driver, as well as because it meant Mac users in the lab were asking me to read files. I thought for sure NI would be rolling out a driver for v2 but after enough waiting I decided to create my own version. I was quite surprised when I was done with writing the code that there was another version already available. I am not sure how I missed this is in my numerous online searches. There are actually multiple versions of the code you mentioned available. The first version I tried I believe was v2 (still available online) and the enumeration of data types was (is?) wrong. That's a HUGE mistake and the code is still posted. I've looked at the other files and I believe with v6 those implementing the code didn't realize that multiple reads of data are possible within a single "header" (for lack of a better word). This means you may write 2000 values to file and only get 100 back, when in reality you need to read the file 19 more times to get all 2000 samples. Version 9 is much better although it doesn't have the same functionality that mine does, particularly the ability to read only a part of a file, or to read interleaved data. At the time of this writing I believe that is the case, obviously this may change.

I think there are a few key points here though:
1) Posting multiple versions of the same file is not a good idea. Versions 2 & 6 shouldn't exist on the file exchange, unless perhaps as links to some website for archival purposes, or with big warning signs saying "This code doesn't work."
2) More importantly, I think this points out the need for better code integration between files on file exchange, both in terms of shared authorship and in terms of the ability to create entry points to data. Imagine if on the Mathworks website we could create a section titled "File processors" and under it there could be a link to TDMS (National Instruments). On that page there could be a short description about the format along with the various solutions that people have offered. In general I think the File Exchange is outdated and not effective. I believe there have been some attempts to write wrappers around the file exchange but that these have been shut down by TMW. Even now I'm moving my new code to github where although it loses its association with the language/program, I feel there is a better chance that someone can contribute to my code and help to make community software.

26 Jun 2012 John F

Ladies and gents, this file is working great. If you're using an older version of Matlab, you'll have to change a couple of the "~" placeholders with what I'm calling "dummy variables" because the ~ is not recognized.

IF ANYONE IS INTERESTED, there is another TDMS conversion program called convertTDMS (v6). I was spinning my tires like mad trying to figure out why that code wasn't working properly.

Unfortunately, convertTDMS would only read part of my file. This one works like a dream.

If anyone has any idea why convertTDMS was only reading part of my file, I'd love to hear your theories.

Best,

JF
Former Noob, Current Novice

25 Jun 2012 Laura

P.S.: the different kinds of data is only a problem when working with the files modified to have double precision. It does work with the original version, but then, with single

25 Jun 2012 Laura

Hi again Jim!,

I continue making use of your TDMS converter (really a good work!) after your successful recommendations to convert the tdms data in double format.

However I have found another small issue that prevents me to successfully convert my new tdms files. I have been taking a look, but again, I have not been able so far to fix it. When trying to run the converter, I find the following error:

??? Attempted to access dataTypes(10); index out of bounds because numel(dataTypes)=3.

Error in ==> TDMS_readFileHelper_v1 at 76
if any(dataTypes ~= dataTypes(10))

Error in ==> TDMS_readTDMSFile at 249
data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

and indeed my tdms files have two types of data: DT_FLOAT, (the only existing one in the first tdms that I was dealing with), but also the last group of data which is now DT_DOUBLE.

What should I do if my tdms file has two types of data? Can I handle it with this converter?

Thanks a lot in advance for whenever you have the time to reply to this request!

Best regards,
Laura

19 Jun 2012 Richard

Thank you very much, Jim.

01 Jun 2012 Brian Kaul  
22 May 2012 Jim Hokanson

Dedric, please check my email. Thanks.
Jim

22 May 2012 Dedric Xu

Hi Jim,
I've encounteered a maddening problem. I have tried to use your function to read my tdms file,but failed. And my file is a little larger, about 60MB.

When I ran your function, it crashed.

The function I used is [output,metaStruct] = TDMS_getStruct(filePath,4);

And the program has ran about 20min,then it crashed.

I gtt the following error:

Warning: Warning: FOR loop index is too large. Truncating to 2147483647.
> In TDMS_readFileHelper_v1 at 66
In TDMS_readTDMSFile at 249
In TDMS_getStruct at 57
Error using TDMS_readFileHelper_v1 (line 184)
Catastrophic error detected, code probably has an error somewhere

Error in TDMS_readTDMSFile (line 249)
data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);

Error in TDMS_getStruct (line 57)
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Is it a Matlab version issue or am I missing something else?

Best regards,
Dedric

25 Apr 2012 Michael Kung  
23 Apr 2012 Laura

It worked perfectly!Thanks a lot for the program and for the fast help!
It is really the best work I have found online for tdms conversion

23 Apr 2012 Laura  
20 Apr 2012 Jim Hokanson

Laura, you are pointing out an issue that I thought would be good to address but that I just haven't gotten to yet. In general I have found it is good to respect the data type so as to prevent memory overflow for a user, but there are obviously times when you want things read in as a double.

I eventually wanted to handle this more gracefully but you can try the following hack to get things to work:
1) in TDMS_initData, remove the string specification from zeros, in your case it sounds like you want case {9 25} to be:
output = zeros(1,nSamples) instead of output = zeros(1,nSamples,'single')
2) in TDMS_getTypeSeekSizes change the read specification to be "type" instead of "type=>type", in your case you want 'single' instead of 'single=>single'

These two changes should make it so that all doubles are read as singles. I eventually want to make a cleaner fix but I've been too busy lately with other things.

20 Apr 2012 Laura

It is some time that I am playing with your routines. It is a fantastic work. However, although it is working fine to convert to matlab a complex .tdms file, I didn't succeed in modifying the precision of the converted values. I guess is specified in the readOptions (different from default), and specifically called in the subroutines TDMS_getTypeSeekSizes and TDMS_readFileHelper_v1. I am not a advanced matlab user. I would really appreciate if you could help me to change those settings.
My problem consists in that I would like to have the record the structured data in double precision instead of in single.
Thanks for your work and attention

LV

19 Apr 2012 Sergiy  
22 Mar 2012 John Vazey

I get an error
??? Error using ==> TDMS_readFileHelper_v1 at 188
Catastrophic error detected, code probably has an error somewhere

Has anyone else had this problem?

JV

01 Dec 2011 Blaine

I'm new to TDMS files I just assumed that they can all be read the same. The test file I have is just an example file I downloaded from NI. I wasn't aware that Raw DAQ MX was different than any other TDMS file.

30 Nov 2011 Jim Hokanson

Regarding improvements in documentation, that is probably one area to improve. Raw DAQ MX data is a proprietary Labview format and I've been turned down in requests to get specifications on that type. I've managed to parse a couple of examples but since I have had so few examples to go off of I haven't tried implementing it. In my experience the raw format is really just read as a normal data type (int16 in my experience??) with properties that specify how to scale that raw data into measured data. I don't know how to extract the real data type from their meta information. I could technically just treat it as some data type, say uint8, but to parse the data generally (not always) relies on knowing the size of the data type. In addition there is some strange buffering of the meta information which I can manually hard code around but I am not sure if it is always present.

That being said, if you have an example file I would be interested in looking at it so that I could eventually offer some workarounds. As I said I have had some luck in extracting the data.

30 Nov 2011 Blaine

I tried to open my large data file with:

[tmp,metaStruct] = TDMS_readTDMSfile('C:\Users\bjmarti4.NDC\Desktop\FIRST_LIGHT_CO2.tdms','GET_DATA_OPTION','getnone')

I get the following error:

??? Error using ==> TDMS_processLeadIn at 50
Currently code is unable to ignore/handle Raw Daq MX data

Error in ==> TDMS_preprocessFile at 179
[flags,info] = TDMS_processLeadIn(fid,lastLetter);

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_exampleFunctionCalls at 95
[tmp,metaStruct] =
TDMS_readTDMSfile('C:\Users\bjmarti4.NDC\Desktop\FIRST_LIGHT_CO2.tdms','GET_DATA_OPTION','getnone')

I was able to open a small test file, but when I try to open my data file is when I get the error

29 Nov 2011 Jim Hokanson

You can be as specific about reading data as you would like. See the documentation file TDMS_retrievingSubsets. You can get as fine grained in reading of the data as you would like, specifying down to the channel and sample numbers if you like. Please let me know if there is anything about the documentation you think should be clarified.

Looking over this more carefully it appears that I have made an error in my documentation related to the meta_struct. I will be updating that soon.

The main error is that subsequent calls is that if you read more more data and want to use the metaStruct from a previous call to reduce parsing time, you still need to pass in the filename again:

TDMS_readTDMSfile(filename,'META_STRUCT',metaStruct,etc)

I tend to recommend using TDMS_getStruct, in which case you would use:

TDMS_getStruct(filename,versionNumber,{'META_STRUCT' metaStruct etc})

29 Nov 2011 Blaine

Is it possible to only read sections of a large (~3.5 GB) TDMS file with your code? I would like to work with pieces of the data at a time. If so, how would I do it?

07 Nov 2011 Jim Hokanson

Andreas, I am unaware of any bugs in the code at this time. Please check the message I sent you.

07 Nov 2011 Andreas

Does the Code work with Matlab 7.11 ?

I've tried it on a 32bit (XP) and a 64bit (Windows7) System with various TDM/TDMS files.
Depending on the type of input file I get 2 different error messages. For native sample files from NI the reaction is:

>> my_tdms_struct = TDMS_getStruct(filePath);
??? Error using ==> TDMS_processLeadIn at 33
Unexpected lead in header

Error in ==> TDMS_preprocessFile at 179
[flags,info] = TDMS_processLeadIn(fid,lastLetter);

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

My self-generated TDMS files (read with NI Diadem and Excel importer without problems) gave

??? Subscripted assignment dimension mismatch.

Error in ==> TDMS_preprocessFile at 456
rawDataInfo(curIndex).dataMatrix(indices,2) = ...

Error in ==> TDMS_readTDMSFile at 223
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Is it a Matlab version issue or am I missing something else?

Best regards,
Andreas
[I'm a different Andreas than the one who posted here before..]

12 Oct 2011 Jim Hokanson

Hi Andreas,

I've known about that bug for a while I just haven't uploaded a fix yet. I had been a bit slow since I wanted to incorporate some other changes and it is an error that will be thrown as opposed to failing silently and pretending to work. I'll upload a modification very soon but for now you just need to replace in that particular case structure (starting around line 100) in TDMS_readFileHelper_v1 the following:
dataType -> change to -> dataTypes(1)

Jim

12 Oct 2011 Andreas

Hi Jim,

It looks like your script does exactly what I need. But when I tried to use it today, I get the following error with my tdms file:

>> test=TDMS_readTDMSFile('test.tdms')

??? Undefined function or variable "dataType".

Error in ==> TDMS_readFileHelper_v1 at 106
temp = fread(fid,numValuesRead*nChans,precisionType{dataType});

Error in ==> TDMS_readTDMSFile at 258
data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);

Reading the index file is working fine and i get the following result:

>>test=TDMS_readTDMSFile('test.tdms_index')

test =
rootIndex: 1
groupIndices: 2
groupNames: {'Analog Input'}
chanIndices: {[3 4 5 6 7]}
chanNames: {{1x5 cell}}
data: {[] [] [] [] [] [] []}
propNames: {{1x1 cell} {1x0 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}}
propValues: {{1x1 cell} {1x0 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell} {1x1 cell}}
objectPathsOrig: {'/' '/'Analog Input'' '/'Analog Input'/'T'' '/'Analog Input'/'U'' '/'Analog Input'/'I'' '/'Analog Input'/'B'' '/'Analog Input'/'H''}
numberDataPointsRaw: [0 0 28000 28000 28000 28000 28000]
dataType: [0 0 10 10 10 10 10]
dataTypeName: {'void' 'void' 'double' 'double' 'double' 'double' 'double'}

Maybe you have a tip for me what I have to change in the script to read my tdms file?

Thank you in advance.

Best regards

Andreas

05 Aug 2011 Jim Hokanson

That shouldn't be a problem. The raw format is currently unsupported because its type is unspecified.

05 Aug 2011 Jeffrey Fletcher

Great piece of work - thanks.
I get an erro when trying to read by TDMS file created directly by the NI-DAQmx .NET TdmsLogging function. The error stack is:

??? Error using ==> TDMS_processLeadIn at 50
Currently code is unable to ignore/handle Raw Daq MX data
Error in ==> TDMS_preprocessFile at 175
[flags,info] = TDMS_processLeadIn(fid,lastLetter);
Error in ==> TDMS_readTDMSFile at 225
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);
Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

It appears that "Raw DAQmx data" is not currently supported. It's too bad because you can create a data logger in MATLAB (with ONLY NI-DAQmx and base MATLAB installed) with a few lines of code. I'll share this with anyone interested.

Any chance of an update that supports this data type?
Jeff

14 Jul 2011 Jim Hokanson

Craig has identified a 2nd bug that is present in v2.2. This bug effects all previous versions. Group names with "/" characters are currently incorrectly parsed. An update to v2.2 will be released soon that addresses this bug, and the getSubset,IgnoreSubset bug. NOTE: Both of these bugs will cause errors instead of failing silently. Thanks guys for letting me know you had a problem so I could improve the code.

Jim

11 Jul 2011 Craig Smith

Jim, Please ignore my earlier comment. I have resolved (stupid user error by me). I now get to the following error:
my_tdms_struct = TDMS_getStruct(filename);
??? In an assignment A(I) = B, the number of elements in B and
I must be the same.

Error in ==> TDMS_readTDMSFile at 255
groupIndices(curGroupCount) = I_groupObject;

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

The value of curGroupCount is 1 but the I_groupObject is an array of length 14 with integer values of 2 through 15. Not sure why the mismatch here

30 Jun 2011 Craig Smith

Jim, I've just downloaded your files and tried to read a TDMS file v2 astarting with what you have in the readMe.txt file. When I run TDMS_getStruct(filename) and select my file I get the following error messages:
??? Undefined function or method 'TDMS_handleGetDataOption' for input arguments of type 'struct'.

Error in ==> TDMS_readTDMSFile at 192
TDMS_handleGetDataOption('check',paramsStruct)

Error in ==> TDMS_getStruct at 57
[temp,metaStruct] = TDMS_readTDMSFile(filePath,readOptions{:});

Any ideas how I should proceed?

13 Jun 2011 Jim Hokanson

Found bug for v2.2 effecting use of GET_DATA_OPTION with values 'getSubset' or 'ignoreSubset'
This bug will already throw an error to let you know the read was incorrect

If you get the error:
The # of requested values does not equal the # of returned values, error in code likely
Then this should be fixed by adding this line:
numValuesToGetActual(~keepDataArray) = 0;

after these lines: (around line 133)
useSubset = false;
subsetInfo = struct([]);
numValuesToGetActual = metaStruct.numberDataPoints;

This will be fixed in the next version.

08 Jun 2011 Jim Hokanson

Hmmm,

In general the code looks fine. I would throw in the already parsed meta structure back into the function call to save processing time. I'll send some instructions on that later. As the code states when a problem like that happens it is most likely my fault. In general I try and throw in checks where possible because I'd rather have things not work and tell you than pretend to work. I'll send you an email with more instructions on how we can find a fix to this problem.

Jim

08 Jun 2011 Juha Suomalainen

Hi,
Great code you have here but I think there is still a bug somewhere inside it. I am able to load my whole TDMS file using you function. That works fine. Unfortunately some of my datasets will be huge and due to memory issues must be loaded and processed in parts. If I try to read a subset of data using your function, I get an error.

Here is my code to read and process data group by group:
---
% Get list what is inside the TDMS
params = {'GET_DATA_OPTION','getnone'};
TDMS_Structure = TDMS_readTDMSFile(TDMS_File, params{:});
% Read and process groups from the TDMS file one by one
for GroupNumber = 1:numel(TDMS_Structure.groupNames)
params = {...
'GET_DATA_OPTION', 'getSubset',...
'OBJECTS_GET', struct('groupsKeep',{{TDMS_Structure.groupNames{GroupNumber}}})};
TempData = TDMS_readTDMSFile(TDMS_File, params{:});
% process subset of data here
ProcessData(TempData)
end
---

I can read the TDMS_Structure without problems, but the subset loading call of 'TDMS_readTDMSFile' produces an error:

---
??? Error using ==> TDMS_readFileHelper_v1 at 211
The # of requested values does not equal the # of returned values, error in code likely

Error in ==> TDMS_readTDMSFile at 229
data = TDMS_readFileHelper_v1(fid,optionStruct,metaStruct,paramsStruct);
---

Is it a bug or am I just using your code with wrong parameters?

07 Jun 2011 Jim Hokanson

Hi Fiona,

Thanks for pointing out the issue. I'm really busy early this week but I might be able to look at this issue Thursday or Friday. See me email for more details.

Jim

07 Jun 2011 Fiona

Thanks for uploading!
When I tried to read my TDMS file, I got the errors below. I wonder if anyone can help. Thanks!

??? Error using ==> TDMS_preprocessFile at 375
The remaining data doesn't split evently into chunks, estimated # of chunks: 5.000000e-001

Error in ==> TDMS_readTDMSFile at 198
metaStruct = TDMS_preprocessFile(fid,tdmsFileName,paramsStruct);

Error in ==> Test at 11
[finalOutput,metaStruct] = TDMS_readTDMSFile(tdmsFileName)

01 Jun 2011 Jim Hokanson

Yes it does. I use it with 64 bit Windows and it has been tested with a 64 bit Mac.

01 Jun 2011 Kevin

please advise if this works with 64bit matlab

27 Apr 2011 Jim Hokanson

@Guosong Zhang
In 2009b the ~ was introduced as a way to ignore outputs. This can be especially useful if that output requires a lot of memory. I've gotten in the habit of putting those in although I realize not everyone may be using 2009b or later. I might upload a fix in which I put dummy variables back, although I'd be tempted instead to write a small script that allows checking the version and doing a replace in all the files if needed.

27 Apr 2011 Guosong Zhang

??? Error: File: TDMS_readTDMSFile.m Line: 453 Column: 11
Expression or statement is incorrect--possibly unbalanced (, {, or [.

25 Mar 2011 Harm

Works great! The NI tool with the DLLs failed on me (64bit version does not seem to work, 32 bit version was OK).
But this one is better; and I think it will work on Linux systems as well.

25 Mar 2011 Harm  
21 Mar 2011 Edward Zechmann  
01 Mar 2011 Ray  
02 Feb 2011 Jim Hokanson

Joacim,

I fixed the unicode bug. Ver. 1.1 should be up soon. I had misread some documentation regarding # of bytes vs # of characters. Thanks for pointing out my mistake.

Jim

25 Jan 2011 Jim Hokanson

Hi all,

In response to Joacim's comments:

1) There should be no file size limitation (other than the users memory size), the out of memory indicates a parsing error.

2) I am still quite unsure as to how well the code handles unicode. The error seems to indicate an ability to properly parse a string ...

3) If anyone sees a similar error please feel free to email an example file to me and I'll gladly try and fix the code so that it is handled.

Thanks,
Jim

21 Jan 2011 Joacim

Impressive amount of work you've put into this. Good job and well documented.
Tested the function on my data saved by Signal Express 2009. Works for small files (<6 MB) but when trying to load larger files (>50 MB) it crashes.
I'm lazy so I downloaded the matlab_tdm_example_sp2010.zip from NI. Using the nilibddc.dll and approx. 200 lines of code, including comments, I'm reading even larger files (approx 320 MB) in less than 4 seconds.

Below is some info from the crash. Maybe it can help you fix the problem:

testTDMSreading
266 propName = fread(fid,propNameLength,'*char')';
??? Error using ==> fread
Out of memory. Type HELP MEMORY for your options.

Error in ==> TDMS_preprocessFile at 266
propName = fread(fid,propNameLength,'*char')';

Error in ==> TDMS_readTDMSFile at 176
[rawDataInfo,numberDataPoints,objectPaths,segInfo] =
TDMS_preprocessFile(fid2,isIndexFID2,params);

Error in ==> testTDMSreading at 34
output = TDMS_readTDMSFile(fullfileName,optionCA{:});

>> testTDMSreading
266 propName = fread(fid,propNameLength,'*char')';
K>> propNameLength
propNameLength =
1.9629e+009
K>> fid
fid =
12
K>> curProps
curProps =
Columns 1 through 8
'format-string' [1x22 char] 'unit_string' [] [] [] [] []
Columns 9 through 20
[] [] [] [] [] [] [] [] [] [] [] []
K>> iProp
iProp =
4
K>> propNameLength
propNameLength =
1.9629e+009

Updates
14 Jan 2011

Updated description

02 Feb 2011

Fixed a major bug with reading unicode. Other small bug fixes and improvements can be seen in TDMS_VERSION_INFO.

19 Mar 2011

Ver. 1.2 - I had an incorrect minus sign when reading the timestamp property which yielded an invalid year. Data timestamp reads were correct and previously tested :)
Thanks to Ed Zechmann for pointing out the problem! Added extra conv. script v3

28 Apr 2011

The main update to version 2 is the ability to retrieve a subset of the data from channels. In addition I've added some extra wrappers to handle data retrieval.

29 Apr 2011

I was a bit quick with the last update (v2.0). It had some bugs with respect to parsing subsets of the data.

17 May 2011

2.2 Update:
Improved speed and specificity with regard to subset handling

15 Jul 2011

Fixed a bug in which an error check I had in place was incorrectly being thrown.
Fixed parsing of object names with ' and / characters.

Also allows for only reading .tdms_index files (for debugging)

12 Oct 2011

Fixed some bugs with interleaved data, as well as a bug in reading timestamp data (not properties) which failed with dates prior to 1904.

30 Jul 2012

Bug fix (potentially silent bug), see version notes file.

Contact us