using fwrite for multiple data type at once

I am trying to write a vector of data of various data types (all numeric) to a .binary file all at once instead of having to loop through the data set. I have a vector that contains the numbers:

myData = [1 3.5 400 75 2.5 6 1];

and then each number has an associated data type:

myDataType = {'single','double','single','single','double','single','single'};

Is there a way to use the fwrite function to write all the data of varying types at once? Such as:

fwrite(fid,myData,myDataType)

This command doesnt work as fwrite. know that for this example that i am giving it would be just as fast to loop through the data and write them to the file one at a time using:

for n = 1:length(myData)
    fwrite(fid,nyData(n),myDataType{n})
end

however the data vector i am writing contains thousands of different data and the format of the data type does not follow any particular set pattern and changes. typically, when all the data is the same such as a vector containing 10,000 double values, it is expoenntially faster to write all of the data at once like this:

fwrite(fid,myData,'double')

than it is to use a for loop. But for my specific case, the data types change. Any help would be greatly appreciated.

Answer 1

[Walter Roberson]

memmapfile() can be used to read and write fixed-length records https://www.mathworks.com/help/matlab/ref/memmapfile.html

Alternately:

myData = [1 3.5 400 75 2.5 6 1];
myDataType = {'single','double','single','single','double','single','single'};

temp = arrayfun(@(val, type) typecast(cast(val, type{1}),'uint8'), myData, myDataType, 'uniform', 0);
bytes = horzcat(temp{:})

bytes = 1×36

0 0 128 63 0 0 0 0 0 0 12 64 0 0 200 67 0 0 150 66 0 0 0 0 0 0 4 64 0 0

Now bytes is a stream of uint8 that can be written.

On x64 architectures, the entries will be in little-endian order. You might potentially want

temp = arrayfun(@(val, type) typecast(swapbytes(cast(val, type{1})),'uint8'), myData, myDataType, 'uniform', 0);
bytes = horzcat(temp{:})

bytes = 1×36

63 128 0 0 64 12 0 0 0 0 0 0 67 200 0 0 66 150 0 0 64 4 0 0 0 0 0 0 64 192

[Eric Alexander]

So after i perform the above lines of code i can simply do:

fwrite(fileID,bytes,'uint8')

and write the data to the binary file? alternatively, when reading that same binary file back, would i do:

val1 = fread(fileID,1,'single');
val2 = fread(fileID,1,'double');
val3 = fread(fileID,1,'single');

and so on? if instead i would have to read the binary file like this:

values = fread(fileId,7,'uint8');

then this solution is not viable as the data is stored in the binary file incorrectly.

[dpb]

You can read it back either way...the latter you would then have to do the cast back from bytes to the float as the reverse of the process that created the byte stream going out.

It's easy-enough to test...

>> fid=fopen('Mixed.bin','rw');
>> fwrite(fid,[typecast(cast(pi,'single'),'uint8'),typecast(cast(pi,'double'),'uint8')]);
>> frewind(fid);
>> fread(fid,1,'single')
ans =
    3.1416
>> fread(fid,1,'double')
ans =
    3.1416
>> fid=fclose(fid);
>> 

[Walter Roberson]

fwrite(fileID,bytes,'uint8')

Yes. And because I constructed bytes to be uint8 you can abbreviate that to

fwrite(fileID, bytes);

[Walter Roberson]

if instead i would have to read the binary file like this: values = fread(fileId,7,'uint8');

That is a valid step, but it is only one step. You then have to divide the byte array into pieces and typecast to the original data type, such as

val1 = typecast(bytes(1:4), 'single');
val2 = typecast(bytes(5:12), 'double');
val3 = typecast(bytes(13:16), 'single');

You can create a function that takes a list of types and does the intepretation:

function [datacell, nextpos] = freadstream(bytes, nextpos, typelist)
   knowntypes = {'single', 'double', 'uint8', 'int8', 'uint16', 'int16', 'uint32', 'int32', 'uint64', 'int64'};
   typebytes = [4, 8, 1, 1, 2, 2, 4, 4, 8, 8];
   nitems = length(typelist);
   datacell = cell(1, nitems);
   buffsize = length(bytes);
   for K = 1 : nitems
       thistype = typelist{K};
       [isknown, typeidx] = ismember(thistype, knowntypes);
       assert(isknown, 'freadstream: unknown item type "%s"', thistype);
       thisbytes = typebtyes(typeidx);
       lastpos = nextpos+thisbytes-1;
       if lastpos > buffsize
           error('ran out of buffer reading entry #%d', K);
       end
       thisvalue = typecast(bytes(nextpos:lastpos), thistype);
       nextpos = lastpos+1;
       datacell{K} = thisvalue;
   end
end

Return would be a cell array of scalar values, and nextpos would be updated to point to the next available position. When first reading from the stream, use nextpos value of 1.

This has obvious extensions to support vectors or arrays, either by making typelist entries possibly being pairs of values including a size, or by encoding the size in characters like '5*double' or '[17,11]*double' . As usual "to end of buffer" could be inf. You could jiggle the behavior at end of stream, like 'eos', 'error' (like it says) vs 'eos', 'whole' (stops processing and returns what it can if stream does not have enough data to satisify an entire request) vs 'eos', 'split' (when an entire request cannot be satisfied, returns as much as is available from it, using the underlying type requested) -- which might require returning a vector instead of reshape according to requested size).

Answer 2

[dpb]

The precision, skip, machinefmt optional inputs to fwrite have not been vectorized (as you have discovered), so you can't intermix the types with array syntax for the output array.

If you write a file in this fashion, however, not only do you have a problem writing it efficiently, you have also created the problem in being able to read it back efficiently--you'll have to read it record-by-record, too, because fread has the same limitation on what is a vector input.

I would strongly suggest to NOT do this at all: I see some possible alternatives:

1. Just write everything as double; if you for some reason your really, really need a given variable to be single, then cast it after reading;
2. Reorder the writing to put all of a given type in one call and the rest in another. This also adds more complication that I can see it possibly be being worth in have to sort things out in both directions again,
3. SAVE the variables to a .mat file instead. Preserves type at the expense of some overhead but much less painful to code and probably at least as fast as the looping solution to do what is requested.

[dpb]

"not only do you have a problem writing it efficiently, you have also created the problem in being able to read it back efficiently--you'll have to read it record-by-record, too, ..."

Actually, it's even worse than that--you have to write/read every array element individually with its corresponding type unless you parse the dataType array and find where the same type is repeated more than once so that you can write those sequential elements together--and that additional logic to keep track of the position and type probably is more expensive than the savings that would be achieved.

[Eric Alexander]

Thank you for the tips. This is a legacy system and I have no control over how the system I'm sending the binary file to operates so changing the format of the binary or the binary to a .mat file is not an option that I have. I've considered writing data sets sequentially based on similar data types but again, the processing time I save doing that is lost in parsing the data into chunks to begin with.

[dpb]

Shoulda' known/guessed.

The memmapfile idea should work; I don't know about performance; the one time I tried using it was on Q? here for large file input and it was much slower (by about 10X) than directly computing the desired location in the file and using fseek to move around. But, that was a case needing to bring in pieces of a very large file; a sequential write operation might be pretty quick.

Alternatively, this is a place where a Fortran mex file might be a handy way to approach it -- the Fortran i/o system would handle the type information transparently on an unformatted WRITE.

Which brings up the Q? of if the legacy system would happen to be a Fortran application, is it actually a stream file (which required nonstandard compiler extensions before F2003) or a direct access file which would also have the hidden record lengths in it.