ConvertTDMS (v9)

Import a LabView TDMS file into the MATLAB workspace

Description

Import a LabView TDMS file (version 1.1 or 1.2) as a structure in the MATLAB workspace with the option to save the data in a MAT file. It was written in MATLAB 2010b. The original function was based on the work by Brad Humphreys of ZIN Technologies and Grant Lohsen & Jeff Sitterle of GTRI (versions 1 through 4 of this function). The Version 9 function (written by Philip Top), in addition to incorporating the various updates that were added to previous versions, can process files that have been "optimized" by LabView. This function has been tested with a limited number of diverse TDMS files.

• MATLAB release: MATLAB 7.11 (2010b)

[Mark Shore]

My suggestion would be to keep the TDMS converter FEX submissions as a single file, with updates, rather than starting a new file number for each new version.

I have not rated it, since when testing with fairly large TDMS files (240 MB) the previous TDMS converter took far longer to read the file (in fact, I halted the process after five minutes) than writing a MATLAB-readable binary data file in LabVIEW and then importing it into MATLAB directly.

Other people's results may differ, of course.

[Michael]

Robert,

Please post a comment on the thread for previous versions of this utility if you are going to post a new version of the ConvertTDMS utility as opposed to an update (I was "watching" the v6 page, but had no idea that you had already posted this v7 until you responded to Chris' post).

Thanks for the utility though. I have had success with it in modified form (making many of the modifications you have done in parallel to you as well as a few of my own). Once I finish making the changes that help me in my particular application, I'd like to coordinate with you so that my changes get wrapped into your latest version.

Mike

[Robert]

Thanks for the suggestion Michael. For some reason, I hadn't thought to do that; it's done now. That's also a good idea to include your modifications in any future updates. I suspect that it'll make the function more robust.

[Robert]

Thanks to André Rüegg for his work in developing the 29 SEP 10 update

[callisto genco]

Hi,

I have download the converter ver 1.7. It is a great job and I see I will save a lot of time.

Unfortunately I have a filename problem:

I have the software Signal Express version 2.5 in french. It has generated files:
Déformation.tdms_index
Déformation.tdms

with the french "é". It gives me the field error "A valid file name could not be created". I changed the tdms datafile in Deformation and I get the same error. I know I can change the file name in the sgnal express software for the next time. Unfortunately I have already generated the sata and it looks like the file name in inside the tdms file as well.
Is there any solution for these problems ?

Regards

Callisto Genco

[Robert]

The special characters that have been identified so far are enumerated in the 'fixcharformatlab' local function (line 767 within version 1.7 updated 29 SEP 10). Add the following line to the bottom of the block of special characters that are already listed within the function: textin=strrep(textin,'é','e'); I intend to determine a more robust method (as opposed to enumeration) to handle the myriad of possible special characters. HTH

[Robert]

The updated file dated 30 SEP 10 hopefully addresses the issue of special characters by identifying characters that are not "A" through "Z", "a" through "z", 0 through 9 or underscore. The characters are replaced with an underscore and a message is written to the Command Window explaining to the user what has happened and how to fix it. This may not be the best solution, but it should be robust. Unfortunately, I was only able to test it with a very limited number of special characters.

[Robert]

Most of the recent errors I have received from users have related in some way to long channel/property names and/or channel/property names that contain special characters (ä, #, ß, é, etc.). These errors happen because the function I used as a basis to write my first version of the current function used those names as the basis for creating structure fieldnames which means they have to adhere to the MATLAB rules for variable names. Accordingly, the main function included a sub-function to create compliant fieldnames. While this technique worked for many data files, it's not very robust and the channel/property names can end-up quite convoluted from their original values. Sometimes it was difficult to correlate fieldnames to channels.

Also, I recently found out that my algorithm to create compliant field names did not work correctly with very long channel names (thanks to Peter Sulcs). It generated the most bothersome kind of error: It didn't trip an actual error, per se; instead, it just concatenated additional data into the wrong channel. So, the user wouldn't necessary know an error had occurred.

In light of all this, I have posted an updated function (posted 12 OCT 10). Now, the sections of the function that create the channel/property names that are used within the body of the function are robust against long channel/property names and names that contain non-UTF8 characters. The original channel/property names are now retained (with no truncation or character replacement) and included in the output structure. I have added the option to return the list of channel names in its own cell array. I have also added a much more detailed 'help' description to the function ('doc convertTDMS'). In the process of modifying the function, I discovered (and corrected) an error in the time stamp calculation (it was off by exactly 1 hour) that appears to have been present since the first version. Unfortunately, I am only able to test the function with a limited number of diverse TDMS files.

[Johan]

Ooops! The rating posted itself before I had the chance to write something.

I really like this function. It is exactly the tool that I need. However, when I try to convert my set of data, it only includes the first 2.000 samples in the mat-file. My original TDMS file contains 300.000 samples per channel and I cannot figure out why it suddenly stops. I have noticed that it loops through 300 segments, each of which contain 1000 samples per channel. After two such turns, the length of the "index vector" reduces to 1 and thereafter, no more samples are read.

Do I have a problem with my TDMS file that convertTDMS cannot interpret or is this due to some bug in the function? I have tried both (v6) and (v8) .

Further, I noticed that the wf_start_time property is not calculated correctly. It differs 6 hours between LabVIEW's TDMS Viewer and the converted mat-file. It would also be nice if it contained fractional seconds.

I would really appreciate if somebody could look into this. I can send my TDMS-file via e-mail or upload it somewhere if that helps.

Regards // Johan

[Jeremie CLEMENT]

Hello,

This function is also very usefull for me. Exactly the tool I need.
But when used to convert a TDMS file coming from a brake dyno, it seems to randomly miss out some values.
So, for example, when an excell TDMS converter will read 1815 values in one block, this function will read only 615values. We are logging at 50Hz, and when analysing the result file, we can see that the time is fine from 0 to 0.54 and then suddenly jump to 1.96... It is the same for all the other channels, missing out random values.

I would really appreciate if someone could help me out.

Thank you for your time.
Best regards
Jeremie

[Robert]

Hi Johan,

You are not the only one who has encountered this issue. I have received emails from several people with what appears to be the same problem (I believe Johan's problem has the same cause). I believe it has to do with how NI applications "optimize" data files that have multiple segments (see the paragraph titled "Optimization" in this NI document: http://zone.ni.com/devzone/cda/tut/p/id/5696). Unfortunately, my function does not appear to properly parse the binary file when the optimization algorithm is invoked. It seems to happen more with TDMS version 4713 files with multiple segments (>10). At this point in time, I'm not sure how to fix the issue. I'm not hopeful that I can fix it anytime soon, especially after reading the last sentence in the "Conclusion" paragraph of the NI document:

"In brief, the TDMS file format is designed to write and read measured data at very high speed, while maintaining a hierarchical system of descriptive information. While the binary layout by itself is rather simple, the optimizations enabled by writing meta data incrementally can lead to very sophisticated file configurations."

If anyone has managed to fix this issue, please let me know.

As far as your date issue, I'm not sure what's going on with the "wf_start_time". I've never investigated it because none of the files I deal with daily store any of the waveform properties. I suspect it has to do with the fact that LabView and MATLAB use different reference years. Also, MATLAB does not have a string form of the date that includes fractions of seconds. You could modify the function to store the date as a number (instead of converting it to "dd-mmm-yyyy HH:MM:SS") which does keep fractions of seconds.

HTH,

Robert

[Robert]

For some reason, the NI link didn't post correctly... http://zone.ni.com/devzone/cda/tut/p/id/5696

[Robert]

Hi Jeremie,

I think you're having the same issue as Johan. Unfortunately, I don't have a solution yet.

Sorry,

Robert

[Robert]

The updated file dated 25 OCT 10 fixes an error with channels that contain no data. Previously, if a channel contained no data, then it was not passed to the output structure even if it did contain properties. It also fixes an error with capturing the properties of the Root object. "GroupNames" was added as an optional output variable. Unfortunately, this update does not address the multi-segment "optimization" issue.

[Johan]

Hi Robert,

Thank you very much for looking into this. The internal layout of a TDMS file seems quite complicated so I understand you're having difficulties. Isn't there anyone at NI who can help us out here? Shouldn't it be of their interest too that there is a well functioning TDMS converter?

I really appreciate your efforts. Regards! /Johan

[Robert]

Hi Johan,

I actually attempted the NI "help desk" route before I decided to write this function. I too thought that NI would want to help. But, that turned-out not to be the case. I basically got referred to a clunky DLL. I did attempt to use it, but it's not robust (i.e. it didn't work with many of my TDMS files), it's very memory intensive and it's very slow. I took some time to try to fix the DLL, but I wasn't able to make it much better. After that, I finally decided to use version 4 of the "convertTDMS.m" function and the web site I posted on 25 OCT 10 to write my own function. There's been a long standing conflict between LabView/NI and The MathWorks because as they've both continued to grow, they have encroached into each other's areas of expertise. My experience has been that neither of those companies are anxious to help users with applications that use their competitors products. I even attempted to get NI help by saying I was writing a C-based application (i.e. not mentioning MATLAB), but even that didn't work. I'm quite sure the expertise to fix my MATLAB function exists within NI, but I doubt I'll ever get access to it. Hopefully, there's a MATLAB user out there that's more knowledgeable than me on the NI end of things that can help make this function more robust.

Cheers,

Robert

[Martin]

Hello Robert,

I'm really pleased to find a tdms program for Matlab, thank you for the effort - it will make my life soo much easier.

At the moment when I try and import this file I get the error below. Is this something you have seen before please?

Thank you,

Martin

Converting '4.tdms'...??? Reference to non-existent field 'cnt'.

Error in ==> convertTDMS>postProcess at 947
if ob.(cname).(cfield).cnt==1

Error in ==> convertTDMS at 796
[ConvertedData(fnum).Data,CurrGroupNames]=postProcess(ob);

[Robert]

Hi Martin,

No, I haven't encountered this error before. If possible, can you please send me a sample data file?

Thanks,

Robert

[Jeremie CLEMENT]

Hi Robert,

Thank you very much for looking into our issue here.
I managed to achieve what I wanted i.e. have our TDMS converted into matfiles, but it is using excel tdms converter from matlab which is much less convenient than your function.

I'll keep an eye on this page to see if any solution comes up as it would be very useful for us.

Thanks again for your time and effort.
Regards,
Jeremie

[Robert]

Hi Jeremie,

At this point, that's probably the most realistic workaround: Launching Excel as a COM object in MATLAB then call the TDMS Add-in function. It will be significantly slower, but at least it will load the data correctly. If I could just get a look at the code within the XLA, I think I could fix the 'convertTDMS.m' function to process optimized files. Unfortunately, that's not possible because it's a compiled application written by NI. Hopefully, I'll be able to eventually get the issue resolved.

Cheers,

Robert

[Robert]

Thanks to some outstanding work from Philip Top, the updated file dated 16 NOV 10 fixes the issue of loading data files that have been "optimized" by LabView. Additionally, he restructured the code to decrease the data loading time.

[Jeremie CLEMENT]

Brillant!

Thank you very much to anyone who was involved in writing this code.

Regards,
Jeremie

[Johan]

Hurray!! The new version works great. Thank you so much Robert and Philip and possibly others who have contributed! /Johan

[Sage Hahn]

 

[Sage Hahn]

This file has been very helpful to me, however a recent format change has generated this error.

Converting 'TestFormat1_345569974.tdms'...??? Subscript indices must either be real positive integers or logicals.

Error in ==> convertTDMS>getChannelInfo at 605
index.(obname).datastartindex(ccnt)=SegInfo.DataStartPosn(segCnt);

Error in ==> convertTDMS at 268
channelinfo=getChannelInfo(fid,SegInfo,NumOfSeg);

[Robert]

Hi Sage,

What do you mean by "... recent format change.. "? Is that a change in how LabView is writing the file, did you change TDMS versions, etc.? If possible, can you please send me a sample data file? That'll probably be the easiest way to try to debug this.

Thanks,

Robert

[Sage Hahn]

recent change in the file I am processing. Not sure what change is causing it. Cannot supply file. Sorry for being 0 help.

[Robert]

Since I can't reproduce the error with any of the sample data files that I have, I'm sorry to say that I can't attempt to fix your issue without a file that does.

[Brad Humphreys]

 

[Brad Humphreys]

First, It's great to see that this tool carried forward. When I wrote this back with version 1.0 tdms files, I thought that I must be creating something others already had but I could not find (never liked the issues that we had to wait on new compiled versions of the .dll when Labview or MATLAB changed).

I did want to pass along one suggestion to those using it. This is actually a LV programming suggestion (hah). A lot of the time when there is complaints about the speed of reading TDMS files it is often related to how the "TDMSWrite" function was called in LV. Every time the write is called, it inserts all of the descriptive header data between the actual raw data. This means that the header data has to be read and parsed. CALL THE TDMSWrite only when you need to in Labview, concat your waveforms or arrays together over several cycles and then TDMSwrite. It will immensely help the speed (and file size). I believe this is what the LV tdms optimizer cleans up. Not for sure, but it looks like there are people here that would know ;)

[Joacim]

Seems like a lot of people are trying to import tdms in Matlab. I have a solution that works for me, reading 320 MB in less than 4 seconds. However, at the moment it requires that the .tdms file is accompanied by a "_meta.txt" file generated by the NI-software.
My question to you: Is this meta-file generated by all NI-software?

[Michael]

I would have preferred to use a native MATLAB solution like the one provided here for importing TDMS data, but ConvertTDMS still doesn't work correctly with all of TDMS files.

For this reason I resorted to using NI's TDMS reading DLL. The script they provide is pretty useless so I wrote my own. Because it uses NI's DLL, it should work properly with all TDMS files (in theory at least).

Check this NI forum page if using a MATLAB script with NI's DLL is a viable solution for you. You are welcome to steal and adapt the script I wrote. http://forums.ni.com/t5/LabVIEW/TDMS-file-will-not-properly-import-into-Matlab/m-p/1415232#M548709

[Pribislav]

Thanks a lot for this great import function and all the work you put into it. However I have a problem reading some data. Most of the TDMS file is read perfectly fine as compared to the Excel-importer result. But in two channels the data are somehow messed up. The TDMS file consists of 42 channels in 4 groups. All but two channels have the same size (approx. 500 values). These two channels have more like 5000 values, but vary in size. The first values in these channels are ok but after approx. 10 values some values are skipped and after approx. 700 values there are only zeros until the end. But the size of the channel is correctly read. These values are usually 4 to 7 digits long and are written as doubles. I run Matlab 2008a on a Windows XP machine. I would be really gratefull for any hint to solve this issue.

Thanks a lot,
Pribislav

[Michael]

Pribislav,

I had a similar issue with some of my TDMS files. I don't know if it is a viable option for you or not, but I wrote a MATLAB script that makes use of NI's DLL to process TDMS files. That wasn't my preferred solution, but it works for my files. See the link in my post from 21 JAN 2011 above.

-Mike

[Pribislav]

Mike,

thanks for your reply. I tried your script already. Unfortunately Matlab crashed completely when running it (Segmentation violation, somehow connected to the m_interpreter.dll). For the ConvertTDMS.m v9 script I could figure out, that some values are just skipped, meaning that all values being read are correct but huge chunks in between are missing. Somehow the datastartindex and/or the multiplier miss some entries.

Thanks, Pribislav

[Robert]

Hi Pribislav,

It sounds like to me that your issue may be caused by interleaved data. Regrettably, dealing with that is out of my realm of expertise. I recommend checking your LabView application that's creating the file to see if it is in fact writing the data with an interleaved layout. Unfortunately, LabView may be doing it automatically (i.e. you can't change it) as part of its undocumented "optimization" algorithm. I suspect that the latter is the case, especially since the Excel TDMS add-in reads the file correctly. Perhaps Philip Top has some recommendations on how to fix your issue.

As you found out, using NI's DLL isn't a very robust solution. Initially, I tried using it to read my TDMS files and it didn't take long for me to abandon that approach and develop this function.

Sorry I can't be more help.

Regards,

Robert

[Pribislav]

Hi Robert & Michael,
first of all thanks to both of you for helping out all those desperate Matlab users. However I found a "solution" to my problem. The function Michael wrote and published at NI works perfectly fine with Matlab 2010b (but not 2008a). I think the problem is somehow due to the fact that my TDMS file contains strings and integers. It is certainly not due to interleaved data, I checked that one (they are not interleaved).

Cheers,
Pribislav

[Robert]

Thanks to Philip Top and Pribislav, the updated function dated 18 MAR 11 fixes several bugs. It should post here soon.

[Sage Hahn]

This update has resolved my previous problems reading test files. Thank you so much!

[Noah]

Works Great. Thanks for the update. Tried to use the NI files alone and it was a mess. This was much cleaner and works very cleanly. Thanks again.

[George Metaxas]

A great routine, worked 100% right away. Thanks Robert!

[Sage Hahn]

With large test files I am getting>

Converting 'testData.tdms' ...??? Error Out of Memory. Type HELP MEMORY for your options.

Error in ==> convertTDMS>getChannelInfo at 584
index.(obname).index = zeros(NumOfSeg,1);

Error in ==> convertTDMS at 292
channelinfo=getChannelInfo(fid,SegInfo,NumOfSeg);

Watching memory usage as file is converted I see huge spike right be for error occurs (obviously).

[Robert]

Thanks to Philip Top, the updated function dated 17 MAY 11 fixes a couple of bugs that arose with NI-DAQmx 9.2.3. It should post here soon. I don't know if it will help with the "out of memory" issue that Sage has encountered.

[Ivy]

Thank you for your contribution. I appreciate it.

In my case, reading 85 MB TDMS file takes 1 hour. And there is an error:
In an assignment A(:) = B, the number of elements in A and B
must be the same.

Error in ==> convertTDMS>postProcess at 1109
                    Value(c)=index.(cname).(cfield).value;

Error in ==> convertTDMS at 287
    [ConvertedData(fnum).Data,CurrGroupNames]=postProcess(ob,channelinfo);

Thank you!

[tony]

 

[Daniele]

Hello. I can't run this application. When I press the start button the following error appears:

??? Error using ==> convertTDMS at 225
Not enough input arguments.

I'm still a dummy user of Matlab....

Can you help me, please? Thanks.

Regards

Daniele

[Daeniel]

You need two input arguments.
e.g.:
a = convertTDMS(0,'filename.tdms');

Zero as the first argument states that you don´t want Matlab to save the converted data automatically. If you give a 1 as the first argument the data will automatically be saved as a .mat- file.

It works for me! Thank you very much. Saves a lot of time.

BR, Daeniel

[Current]

Hi, thank you very much for developing this code. I am having some trouble extracting the data from my TDMS files. After conversion, there will be a 1x1 structure with "FileName", "FileFolder", "SegTDMSVerNum", "NumOfSegments", and "Data". However, none of these values contain the actual measurement data. Under "Data", it only lists the names of the channels measured from my data acquisition board (under "MeasuredData", and the "Root".

Does this code allow you to be able to extract the measured data? Or only the structure of the TDMS file.

Thanks!

[Daniel]

Many thanks to all who made this possible. It is a huge improvement over the DLLs that NI publishes.

[Xiao]

Hi all, I got same issue as Current got. convetTDMS() works OK, but created mat file can not retrive measured data, only structure. Did I miss something?

[Robert]

Hi Xiao,

Your data should be contained within the structure.

ConvertedData=convertTDMS(SaveFlag,FileName);

To access the data, iterate through the entries within the "ConvertedData.Data.MeasuredData" field. This field contains 4 fields: "Name" (string), "Data" (numeric), "Total_Samples" (numeric) and "Property" (structure). The data are contained within the "ConvertedData.Data.MeasuredData.Data" field. Here's one way to see all the channels and their number of samples in the Command Window and to retrieve the data for a particular channel of interest (you need to specify the "ChanNameOfInterest").

for x = 1:numel(ConvertedData.Data.MeasuredData)

fprintf('\nData set %.0f is %s and contains %.0f samples.\n',x,ConvertedData.Data.MeasuredData(x).Name,ConvertedData.Data.MeasuredData(x).Total_Samples)

if strcmpi(ChanNameOfInterest,ConvertedData.Data.MeasuredData(x).Name)
MyData=ConvertedData.Data.MeasuredData(x).Data
end

end

If this method doesn't work, then NI may have changed their file format (again). If all else fails, you should be able to find a free TDMS reader as an Excel add-in.

HTH,

Robert

[Dedric Xu]

Hi,Robert
Now I am encounteering a maddening problem, which is that my TDMS file format version is 2.0. And I have tried to use your function to read this file,but failed.

The command window says "Reference to non-existent field 'rawdataindex'.
Error in convertTDMS_one (line 414)
index(end).rawdataindex=index(end-1).rawdataindex;"

Could you help me solve this problem?
Thank you very much.

[Robert]

Hi Dedric,

I’m sorry but I won’t be any help with your problem. It’s been well over a year since I’ve processed any TDMS files so, I no longer work with this function nor track any LabView/NI changes. Perhaps some others that monitor this function may be able to help you. I know there are a few that have helped me in the past to get it to the point where it is now.

Good Luck,

Robert

[Brad Humphreys]

On a quick look, it looks like this could happen when there is no root object in the TDMS file. Have does the file look when you open it in Labview using their TDMS viewer? There should be a "file tree". the first level of the tree will be "/" and then there will be objects under it. Is that how your file looks?

By the way, how big is the file in bytes?