portable matfile exporter (in C)

Export matfiles from C programs, regardless of the platform the file is running on.

Description

Sometimes it is convenient to export data as a binary matfile, so that it can be manipulated in Matlab. So here are two convenience functions to write matrices and strings. They don't rely on any Matlab libraries being available.

If your floating point unit is not ieee754 then the bits cannot simply be written out, so a portable floating point save routine has been provided. It handles NaNs, infinity, and denormalized numbers. So you can export mat files directly from a supercomputer running weird hardware, and you can write ANSI C programs knowing they will work correctly on every ANSI C compiler.

• MATLAB release: MATLAB 7.9 (2009b)
• Other requirements: ANSI C compiler.

[James Tursa]

This is a nifty piece of code that has potential. I got small examples to work on a PC and on an ALPHA mainframe. Both files loaded into MATLAB correctly. However, I would like to see the following improvements:

- More comments in the code. There are *some* comments to give the reader a clue as to what is being written to the file, but not enough to allow any insight that would help modification.

- An input for the global flag. As it is, everything read gets put in the global workspace. It would be better if that was a user input.

- Support for multi-dimensional arrays.

- Support for complex arrays.

- Support for multiple variables per file. This is a severe limitation to this code right now. If I have 30 variables to port over to MATLAB I have to create 30 different files. Very cumbersome. It would be better to put them all in one file.

- More error checking on the inputs. For example, if name is too long then the code will write beyond the end of the buff array. There is no check for this condition.

- The x != x test for NaN could be turned into a function call to avoid the compiler optimization from deleting the code. Maybe put in a compile option to do this instead of the current test.

- If the data is already in column (i.e., MATLAB) order, no need for a double for loop to write the elements out. A simple linear scan through the array will suffice.

- Missing prototype for fput16le function.

- Does not preserve NaN bit patterns. (They stay NaN, but the mantissa bits are potentially different). Probably not a big deal, but maybe put a comment in the code notifying the user of this.

- Very slow for large matrices. Probably because of the large number of floating point operations involved in the conversion to IEEE format. Maybe put in some smart checks to detect IEEE input and then skip the conversions altogether in these cases. Or put in some bit level manipulation code instead of the floating point code.

- Not sure if I am using the code incorrectly or not yet, but I am getting some errors. On very small matrix sizes, things seem to work. But for even moderate sizes (e.g., 50 x 1) I am getting some data corruption. It appears that extra bytes are randomly appearing in the file and shifting the floating point data, so subsequent loading produces garbage results. And on larger matrix sizes (e.g., 1000 x 1) I sometimes can't even load the file (I get a corrupt mat file error message from MATLAB). Again, I don't know if I am misusing the code or not yet, but thought I would mention the potential problem.

I will withhold a rating until the author can address my comments.

[Malcolm McLean]

I can't replicate your bug. I'm saving and loading a 50x1 matrix with no problems.
Which version of Matlab do you have?

[James Tursa]

Still haven't figured out the problem yet. I did the following:

- Put the prototype for fput16le at the top of the file.

- Changed your double matrix to be arbitrary size depending on a defined value of N and filled it with the values 1 ... N.

- Commented out the cell stuff so I am only working with the double matrix.

Sometimes things work and sometimes they don't. When I go back & forth with N=5, N=10, N=12, N=10, etc. I typically get the "can't load" error on the N=10. I can't fault your code yet since I don't know what the problem is. Still looking into it. I am using R2007a on 32-bit WinXP.

[James Tursa]

The problem turned out to be the write mode of the file. Instead of opening with "w", open with "wb" to force a binary file open instead of a text file open. I have no idea why the size of the array being written makes a difference when reading the file in "w" mode, but I consistently got everything to work with N=1 ... 20 except for the N=10 case, which always failed to load. On Unix I guess the b option doesn't make a difference if I understand the doc correctly, but on Windows it does.

I practically pulled my hair out over this one. I wrote exactly the same mat file directly under MATLAB (same variable name, size, values, etc.) and then generated the same thing with a modified version of your program. I even went so far as to copy the exact header text of the MATLAB generated mat file into your code. I compared the files and they matched byte for byte ... but the mat file generated by the C code would not load. I thought I was losing my sanity for a moment until I noticed the "w" vice "wb".

Side note: MATLAB stores the tag info differently for the Array Name when the name length is 4 or less. Since it can fit the whole thing in an 8-byte space (a requirement for downstream data alignment), it uses 2-byte integers for the type and bytes (instead of the usual 4 bytes each), and 4 bytes for the name itself. I couldn't find any reference to this in the MATLAB Level 5 MAT File Format doc and it caused some confusion as well. (Why on earth would you change the file format in a confusing manner like this, and not document it, all to save < 8 bytes ???!!! I've got some choice words for *that* designer!) Anyway, I think I figured it out and am much happier now. Did you run across any other gotchas like this that were not documented or poorly documented?

[James Tursa]

I need to retract my statement about the short ints being used in some cases as not being documented. This format *is* in fact documented under the section "Small Data Element Format" ... it is just not documented under the individual sub-element sections.

[Malcolm McLean]

Duh. Rank beginner's error. Of course the file must be opened with "wb".
The problem is that on some platforms it makes no difference. Where it does, you'll only see errors when writing line feeds. So that's as why I couldn't replicate your bug.
 I've uploaded a fix.
I'll address ypur other points later.

[Kevin Shaw]

The author's pejorative comment is completely unnecessary.