File size continuously increase when use h5D.write() to update some data block in original place.

11 views (last 30 days)
Hi, I am trying to update a data block into a h5 file using h5D.write.
The updated file opens with no problem, but the file size continuously increased about 6MB everytime I update.
The code is below:
% the data block size is [sizeX,sizeY,1], always at the same position.
fid = H5F.open(aPath,'H5F_ACC_RDWR','H5P_DEFAULT');
dset_id = H5D.open(fid,['/Data']);
block = [sizeX,sizeY,1];h5_block = fliplr(block); %the same size as original file
dims = [sizeX,sizeY,1];
mem_space_id = H5S.create_simple(length(dims),h5_block,[]);
space_id = H5D.get_space(dset_id);
aStart = [0,0,0];h5_start = fliplr(aStart);
H5S.select_hyperslab(space_id,'H5S_SELECT_SET',h5_start,[],[],h5_block);
H5D.write(dset_id,'H5ML_DEFAULT',mem_space_id,space_id,'H5P_DEFAULT',aData); %aData is updated data of same size as original file
H5D.close(dset_id);H5F.close(fid);
Since no size expansion is involved, Is there a way to avoid file size increase?

Accepted Answer

Walter Roberson
Walter Roberson on 13 Dec 2021
Sorry, no.
When you update, HDF5 writes a new block in, and then marks the old block as no longer in use.
Unfortunately it also has no defined garbage collection, so although the next time around the now-unused block would in theory be large enough to store the data to be written, in practice it may not be.
There is no inherent HDF5 operation to reclaim space.
  3 Comments
Walter Roberson
Walter Roberson on 13 Dec 2021
If the data is being compressed, then remember that a change in value may result in the compressed data being shorter or longer than the previous version.
h5repack appears to support a command line argument to indicate compression type.
For example -f GZIP=9

Sign in to comment.

More Answers (0)

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!