Clear Filters
Clear Filters

How to read xml file with binary data into Matlab? (VTK/VTU File)

13 views (last 30 days)
First of all, I've also asked this question on another site but not had any luck, so I thought I'd try here too. I'll cross-post any answer to either site.
I would like to be able to read in an xml formal file which has a section containing binary data. An example file is shown below:
<?xml version="1.0"?>
<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian">
<UnstructuredGrid>
<Piece NumberOfPoints="1941" NumberOfCells="11339">
<PointData>
<DataArray type="Float64" Name="magnetic field strength" NumberOfComponents="3" format="appended" offset="0"/>
<DataArray type="Float64" Name="magnetic flux density" NumberOfComponents="3" format="appended" offset="46588"/>
<DataArray type="Float64" Name="magnetic vector potential" NumberOfComponents="3" format="appended" offset="93176"/>
</PointData>
<CellData>
<DataArray type="Int32" Name="GeometryIds" format="appended" offset="139764"/>
</CellData>
<Points>
<DataArray type="Float64" NumberOfComponents="3" format="appended" offset="185124"/>
</Points>
<Cells>
<DataArray type="Int32" Name="connectivity" format="appended" offset="231712"/>
<DataArray type="Int32" Name="offsets" format="appended" offset="403396"/>
<DataArray type="Int32" Name="types" format="appended" offset="448756"/>
</Cells>
</Piece>
</UnstructuredGrid>
<AppendedData encoding="raw">
_XF@Loû1q@`@!?V7^W@9DCz@bd@Yb@r <snip>
</AppendedData>
</VTKFile>
This is a VTK data file, specifically the unstructured gid type, for which the .vtu extension is used. The format of this is normal xml, but with a section AppendedData where there is an underscore followed by binary data, the xml describes where each of the data sequences start and end in this data.
Matlab's xmlread can't read this file, I presume because of the binary portion. I get the error below:
[Fatal Error] elmer_3d_magnet_mesh.dat0001.vtu:24:1: Invalid byte 1 of 1-byte UTF-8 sequence.
Error using xmlread (line 97)
Java exception occurred:
org.xml.sax.SAXParseException; systemId: file:/home/rcrozier/Sync/cad_models/elmer_3D_magnet/elmer_3d_magnet_mesh/elmer_3d_magnet_mesh.dat0001.vtu; lineNumber: 24;
columnNumber: 1; Invalid byte 1 of 1-byte UTF-8 sequence.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
However, I can successfully read in the xml portion of the file (using fgetl to read up to the AppendedData tag). I can then create a temporary xml file by adding the missing closing tags and reading this in using xmlread. I can then parse the xml to determine the data structure. This just leaves the reading in the binary portion part. At the end of reading the xml data fgetl leaves me at the file position corresponding to the start of the line with the underscore.
How can I ignore the underscore character, then read in the binary data?
Actually it is the 'ignoring the underscore character' part that is proving difficult as I can't figure out out how to do this without knowing about the character encoding of the file (file -bi returns application/xml; charset=binary on one example).
In case it's of interest, the actual vtk file format specification can be found here (pdf)
Below is the code to get the first text xml part of the file with fgetl
% open the file
fid = fopen(filename, 'r');
% close file when we're done
CC = onCleanup (@() fclose(fid));
xmlstrs = {fgetl(fid)};
find = 1;
while ischar (xmlstrs{find})
find = find + 1;
xmlstrs{find,1} = fgetl(fid);
if ~isempty(strfind (xmlstrs{find,1}, 'AppendedData'))
xmlstrs = [ xmlstrs; {'</AppendedData>'; '</VTKFile>'} ];
% could get file position like this? how many bytes?
datapos = ftell (fid) + 4;
break;
end
end

Accepted Answer

Richard Crozier
Richard Crozier on 5 Aug 2016
The answer to determining the position was to calculate the character bit length from the first line read in like so:
% open the file
fid = fopen(filename, 'r');
% close file when we're done
CC = onCleanup (@() fclose(fid));
xmlstrs = {fgetl(fid)};
firstlinebytes = ftell (fid) - 1;
bytesperchar = round (firstlinebytes / numel (xmlstrs{1}));
then the position of the first byte in the data section is
datapos = ftell (fid) + bytesperchar;
Note, that this isn't the whole answer to reading 'raw' type data in the AppendedData section which is poorly documented. You will find more info on the format of 'raw' (rather than 'base64') data here, but the short answer is it's encoded like the following:
_NNNN<data>NNNN<data>NNNN<data>
^ ^ ^
1 2 3
where each "NNNN" is an unsigned 32-bit integer, and <data> consists of
a number of bytes equal to the preceding NNNN value. The corresponding
DataArray elements must have format="appended" and offset attributes
equal to the following:
1.) offset="0"
2.) offset="(4+NNNN1)"
3.) offset="(4+NNNN1+4+NNNN2)"
  4 Comments
ABDEL MOUMEN
ABDEL MOUMEN on 26 May 2019
Edited: ABDEL MOUMEN on 26 May 2019
Hi Richard,
Thanks for your time and for sharing,
I used the function cited belllow to extract a scalar field from pvtu file but it returns the following error :
Error using str2num (line 35)
Input must be a character vector or string scalar.
Error in read_vtkpoly>get_int (line 48)
myint = str2num(regexp(fline{line_nb},regex_str,'match','once'));
Error in read_vtkpoly (line 23)
num_vertices = get_int(fline2,'NumberOfPoints');
Could you please help me to fix that?
Thanks again

Sign in to comment.

More Answers (1)

Zakia Tasnim
Zakia Tasnim on 18 Oct 2018
where should i put the file name?
  2 Comments
Tiago Pestana
Tiago Pestana on 20 Oct 2018
function [v,f,s] = read_vtkpoly(fname,scalar_name)
v are the verticies f the faces s is the scalar field you want to retrieve
fname is the file name scalar_name is the variable name to be read

Sign in to comment.

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!