File size continuously increase when use h5D.write() to update some data block in original place.

Hi, I am trying to update a data block into a h5 file using h5D.write.

The updated file opens with no problem, but the file size continuously increased about 6MB everytime I update.

The code is below:

% the data block size is [sizeX,sizeY,1], always at the same position.
fid = H5F.open(aPath,'H5F_ACC_RDWR','H5P_DEFAULT');
dset_id  = H5D.open(fid,['/Data']);
block = [sizeX,sizeY,1];h5_block = fliplr(block); %the same size as original file
dims = [sizeX,sizeY,1];
mem_space_id = H5S.create_simple(length(dims),h5_block,[]);
space_id = H5D.get_space(dset_id);
aStart = [0,0,0];h5_start = fliplr(aStart);
H5S.select_hyperslab(space_id,'H5S_SELECT_SET',h5_start,[],[],h5_block);
H5D.write(dset_id,'H5ML_DEFAULT',mem_space_id,space_id,'H5P_DEFAULT',aData); %aData is updated data of same size as original file
H5D.close(dset_id);H5F.close(fid);

Since no size expansion is involved, Is there a way to avoid file size increase?

Answer 1

[Walter Roberson]

Sorry, no.

When you update, HDF5 writes a new block in, and then marks the old block as no longer in use.

Unfortunately it also has no defined garbage collection, so although the next time around the now-unused block would in theory be large enough to store the data to be written, in practice it may not be.

There is no inherent HDF5 operation to reclaim space.

There are some tools that will repack HDF5 files; see https://www.mathworks.com/matlabcentral/answers/395920-how-can-i-delete-a-dataset-completely-from-a-group-in-a-hdf5-file

[raym]

Yes, I use C:\Program Files\HDF_Group\HDF5\1.8.19\bin\h5repack.exe to repack and the file size is reduced now.

[raym]

but another problem is that the increased file size is much larger than original file size.

It may be caused by the different deflate option.

As this is re-write data block instead of create a data set from begining, Is there a way to indicate the deflate option to updated data?

[Walter Roberson]

If the data is being compressed, then remember that a change in value may result in the compressed data being shorter or longer than the previous version.

h5repack appears to support a command line argument to indicate compression type.

https://www.systutorials.com/docs/linux/man/1-h5repack/

For example -f GZIP=9