Thread Subject: reading HDF5 dataset subset

Using the low-level HDF5 functions in Matlab, I have generated some HDF5 files with several datasets that are each 512x512x1410. I need to be able to read subsets of these different layers. Looks like hdf5read does not have subsetting capabilities like hdfread, so I am guessing I need to use low-level functions such as H5D.read. How do I translate something like the following high level hdfread call into those HDF5 low-level functions?:

data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]});
e.g.
sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]});

(Chunk size is set to 512x512x1, but I need to read across those boundaries as well.)

Thanks very much,
Tess

On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon@noaa.gov> wrote:

> Using the low-level HDF5 functions in Matlab, I have generated some HDF5
> files with several datasets that are each 512x512x1410. I need to be
> able to read subsets of these different layers. Looks like hdf5read
> does not have subsetting capabilities like hdfread, so I am guessing I
> need to use low-level functions such as H5D.read. How do I translate
> something like the following high level hdfread call into those HDF5
> low-level functions?:
>
> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]});
> e.g.
> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]});
>
> (Chunk size is set to 512x512x1, but I need to read across those
> boundaries as well.)
>
> Thanks very much,
> Tess

Tess,

http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/

HDF5 (1.6)
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m

On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama
<first.last@mathworks.com> wrote:

> On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon@noaa.gov> wrote:
>
>> Using the low-level HDF5 functions in Matlab, I have generated some
>> HDF5 files with several datasets that are each 512x512x1410. I need to
>> be able to read subsets of these different layers. Looks like hdf5read
>> does not have subsetting capabilities like hdfread, so I am guessing I
>> need to use low-level functions such as H5D.read. How do I translate
>> something like the following high level hdfread call into those HDF5
>> low-level functions?:
>>
>> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]});
>> e.g.
>> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]});
>>
>> (Chunk size is set to 512x512x1, but I need to read across those
>> boundaries as well.)
>>
>> Thanks very much,
>> Tess
>
> Tess,
>
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/
>
> HDF5 (1.6)
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m
>
>

On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama
<first.last@mathworks.com> wrote:

> On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon@noaa.gov> wrote:
>
>> Using the low-level HDF5 functions in Matlab, I have generated some
>> HDF5 files with several datasets that are each 512x512x1410. I need to
>> be able to read subsets of these different layers. Looks like hdf5read
>> does not have subsetting capabilities like hdfread, so I am guessing I
>> need to use low-level functions such as H5D.read. How do I translate
>> something like the following high level hdfread call into those HDF5
>> low-level functions?:
>>
>> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]});
>> e.g.
>> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]});
>>
>> (Chunk size is set to 512x512x1, but I need to read across those
>> boundaries as well.)
>>
>> Thanks very much,
>> Tess
>
> Tess,
>
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/
>
> HDF5 (1.6)
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m
>
>

Sorry, that went out before I could complete it.

The above examples are for MATLAB R2009b and earlier, it would help to
have a quick look at
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/README.txt
to figure out the correct way to specify the start/stride/count arguments.

Please use the 1.8 link for 10a.

You could look at the checksum example for specifying chunking.

HTH

Thanks; I've seen these examples but haven't been able to get them to apply to my problem successfully. Is 'hyperslab' just a fancy word for subset or chunk of the data? Does it matter at all how the data set is chunked?

I ran the following code and received the following error:
fileID=H5F.open(file);
dset=H5D.open(fileID,'/WeeklySST'); % this is a 53x512x512 data set which I
% read successfully as a whole using H5D.read
space=H5D.get_space(dset);
start=[0 0 0];
stride=[50 50 50] % making it symmetrical b/c I'm still figuring out how HDF5 flips
% dimensions
count=[]; block=[]; % "If count and block are specified as [], the count and block
% size default to a single element."
H5S.select(space,'H5S_SELECT_SET',start,stride,count,block);

??? Error using ==> hdf5lib2
The HDF5 library encountered an error: "unable to set hyperslab selection"

Error in ==> H5S.select_hyperslab at 35
H5ML.hdf5lib2('H5Sselect_hyperslab', spaceID, op, start, stride, count, block);

I am running 7.8.0 R2009a.

Thanks again,
Tess

"Ashish Uthama" <first.last@mathworks.com> wrote in message <op.u2tjkwbva5ziv5@uthamaa.dhcp.mathworks.com>...
> On Tue, 03 Nov 2009 07:22:26 -0500, Ashish Uthama
> <first.last@mathworks.com> wrote:
>
> > On Mon, 02 Nov 2009 17:15:21 -0500, Tess <tess.brandon@noaa.gov> wrote:
> >
> >> Using the low-level HDF5 functions in Matlab, I have generated some
> >> HDF5 files with several datasets that are each 512x512x1410. I need to
> >> be able to read subsets of these different layers. Looks like hdf5read
> >> does not have subsetting capabilities like hdfread, so I am guessing I
> >> need to use low-level functions such as H5D.read. How do I translate
> >> something like the following high level hdfread call into those HDF5
> >> low-level functions?:
> >>
> >> data=hdfread('filename','/dataset_name','Index',{[start],[stride],[size]});
> >> e.g.
> >> sst=hdfread(file,'/WeeklySST','Index',{[1 1 999],[1 1 1],[512 512 1]});
> >>
> >> (Chunk size is set to 512x512x1, but I need to read across those
> >> boundaries as well.)
> >>
> >> Thanks very much,
> >> Tess
> >
> > Tess,
> >
> > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/
> >
> > HDF5 (1.6)
> > http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/h5ex_d_hyper.m
> >
> >
>
> Sorry, that went out before I could complete it.
>
> The above examples are for MATLAB R2009b and earlier, it would help to
> have a quick look at
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/matlab/HDF5_M_Examples-1.6/README.txt
> to figure out the correct way to specify the start/stride/count arguments.
>
> Please use the 1.8 link for 10a.
>
> You could look at the checksum example for specifying chunking.
>
> HTH

On Tue, 03 Nov 2009 14:11:03 -0500, Tess <tess.brandon@noaa.gov> wrote:

> Thanks; I've seen these examples but haven't been able to get them to
> apply to my problem successfully. Is 'hyperslab' just a fancy word for
> subset or chunk of the data? Does it matter at all how the data set is
> chunked?
>
> I ran the following code and received the following error:
> fileID=H5F.open(file);
> dset=H5D.open(fileID,'/WeeklySST'); % this is a 53x512x512 data set
> which I
> % read successfully as a whole using H5D.read
> space=H5D.get_space(dset);
> start=[0 0 0];
> stride=[50 50 50] % making it symmetrical b/c I'm still figuring out how
> HDF5 flips
> % dimensions
> count=[]; block=[]; % "If count and block are specified as [], the count
> and block
> % size default to a single element."
> H5S.select(space,'H5S_SELECT_SET',start,stride,count,block);
>
> ??? Error using ==> hdf5lib2
> The HDF5 library encountered an error: "unable to set hyperslab
> selection"
>
> Error in ==> H5S.select_hyperslab at 35
> H5ML.hdf5lib2('H5Sselect_hyperslab', spaceID, op, start, stride, count,
> block);
>
> I am running 7.8.0 R2009a.
>
> Thanks again,
> Tess

1.
As I understand it, chunking is defined on the internals of the file, i.e
how the data is stored/organized internally in the file. This appears to
be something required to enable 'extendible' data sets.
http://www.zib.de/zibdoc/zib/mpp/prog/hdf5/Tutor/extend.html

A hyperslab is what you appear to think of as a 'chunk of data'. It is a
subset of the data. I dont think the chunking of the file matters when you
are reading a hyperslab.

I am sure there is more info somewhere in the HDF documentation.

2. Using [] results in the arguments defaulting to a single element. Thats
not what you want if you want to select a single element (look at code
below)

3. You need to use H5S.select_hyperslab

4. You need to specify a memory space ID which matches the size of the
data selected (see code)



Heres something which might help you understand better and get started
with your own code:



%hdf5write('o.h5','/DS1',reshape(1:53*512*512,[53 512 512])); %used to
create sample data, each element ought to be unique


fileID=H5F.open('o.h5','H5F_ACC_RDONLY','H5P_DEFAULT');
dset=H5D.open(fileID,'/DS1'); % this is a 53x512x512 data set
space=H5D.get_space(dset);


start=[7 18 26]; %zero based!
stride=[1 1 1];
count=[1 1 1]; block=[ ];


H5S.select_hyperslab(space,'H5S_SELECT_SET',fliplr(start-1),... %-1 to
take care of 0 based indexing
     fliplr(stride),fliplr(count),fliplr(block)); %fliplr
to take care of row/column major reversal (please see the ReadMe.txt in my
previous post)
mem_space = H5S.create_simple(3, fliplr(count), []); %see Readme.txt for
explanation on need for flipping


rdata = H5D.read(dset,'H5T_NATIVE_INT',mem_space,space,'H5P_DEFAULT');

H5D.close(dset);
H5S.close(space);
H5S.close(mem_space);
H5F.close(fileID);

disp(rdata)


%Cross check with the expected element:
r=hdf5read('o.h5','/DS1');
r(7,18,26)

I think you hit on it with #4 -- I didn't know how to set the memory space.

Apologies for accidentally leaving out the "_hyperslab" in "H5S.select_hyperslab" when copying from my workspace.

Just to be clear, I already wrote these files to be chunked -- that's not the issue. The issue is whether a file needs to be written with hyperslabs in order to use H5S.select_hyperslab, etc. to read hyperslabs? Should I be using H5S.select_elements instead if I wrote the dataset to the file in one piece originally?

"Ashish Uthama" <first.last@mathworks.com> wrote in message <op.u2t4zlc7a5ziv5@uthamaa.dhcp.mathworks.com>...
> On Tue, 03 Nov 2009 14:11:03 -0500, Tess <tess.brandon@noaa.gov> wrote:
>
> > Thanks; I've seen these examples but haven't been able to get them to
> > apply to my problem successfully. Is 'hyperslab' just a fancy word for
> > subset or chunk of the data? Does it matter at all how the data set is
> > chunked?
> >
> > I ran the following code and received the following error:
> > fileID=H5F.open(file);
> > dset=H5D.open(fileID,'/WeeklySST'); % this is a 53x512x512 data set
> > which I
> > % read successfully as a whole using H5D.read
> > space=H5D.get_space(dset);
> > start=[0 0 0];
> > stride=[50 50 50] % making it symmetrical b/c I'm still figuring out how
> > HDF5 flips
> > % dimensions
> > count=[]; block=[]; % "If count and block are specified as [], the count
> > and block
> > % size default to a single element."
> > H5S.select(space,'H5S_SELECT_SET',start,stride,count,block);
> >
> > ??? Error using ==> hdf5lib2
> > The HDF5 library encountered an error: "unable to set hyperslab
> > selection"
> >
> > Error in ==> H5S.select_hyperslab at 35
> > H5ML.hdf5lib2('H5Sselect_hyperslab', spaceID, op, start, stride, count,
> > block);
> >
> > I am running 7.8.0 R2009a.
> >
> > Thanks again,
> > Tess
>
> 1.
> As I understand it, chunking is defined on the internals of the file, i.e
> how the data is stored/organized internally in the file. This appears to
> be something required to enable 'extendible' data sets.
> http://www.zib.de/zibdoc/zib/mpp/prog/hdf5/Tutor/extend.html
>
> A hyperslab is what you appear to think of as a 'chunk of data'. It is a
> subset of the data. I dont think the chunking of the file matters when you
> are reading a hyperslab.
>
> I am sure there is more info somewhere in the HDF documentation.
>
> 2. Using [] results in the arguments defaulting to a single element. Thats
> not what you want if you want to select a single element (look at code
> below)
>
> 3. You need to use H5S.select_hyperslab
>
> 4. You need to specify a memory space ID which matches the size of the
> data selected (see code)
>
>
>
> Heres something which might help you understand better and get started
> with your own code:
>
>
>
> %hdf5write('o.h5','/DS1',reshape(1:53*512*512,[53 512 512])); %used to
> create sample data, each element ought to be unique
>
>
> fileID=H5F.open('o.h5','H5F_ACC_RDONLY','H5P_DEFAULT');
> dset=H5D.open(fileID,'/DS1'); % this is a 53x512x512 data set
> space=H5D.get_space(dset);
>
>
> start=[7 18 26]; %zero based!
> stride=[1 1 1];
> count=[1 1 1]; block=[ ];
>
>
> H5S.select_hyperslab(space,'H5S_SELECT_SET',fliplr(start-1),... %-1 to
> take care of 0 based indexing
> fliplr(stride),fliplr(count),fliplr(block)); %fliplr
> to take care of row/column major reversal (please see the ReadMe.txt in my
> previous post)
> mem_space = H5S.create_simple(3, fliplr(count), []); %see Readme.txt for
> explanation on need for flipping
>
>
> rdata = H5D.read(dset,'H5T_NATIVE_INT',mem_space,space,'H5P_DEFAULT');
>
> H5D.close(dset);
> H5S.close(space);
> H5S.close(mem_space);
> H5F.close(fileID);
>
> disp(rdata)
>
>
> %Cross check with the expected element:
> r=hdf5read('o.h5','/DS1');
> r(7,18,26)
>
>

I just ran your code and realized that while you defined the memory space to be rank 3, [x y z], the output is only one value. The way you've defined start, stride and count, you should have ended up with a 1x3 vector. I need to be able to subset, say, a [1x512x512] slice of data.

"Ashish Uthama" <first.last@mathworks.com> wrote in message <op.u2t4zlc7a5ziv5@uthamaa.dhcp.mathworks.com>...
> On Tue, 03 Nov 2009 14:11:03 -0500, Tess <tess.brandon@noaa.gov> wrote:
>
> > Thanks; I've seen these examples but haven't been able to get them to
> > apply to my problem successfully. Is 'hyperslab' just a fancy word for
> > subset or chunk of the data? Does it matter at all how the data set is
> > chunked?
> >
> > I ran the following code and received the following error:
> > fileID=H5F.open(file);
> > dset=H5D.open(fileID,'/WeeklySST'); % this is a 53x512x512 data set
> > which I
> > % read successfully as a whole using H5D.read
> > space=H5D.get_space(dset);
> > start=[0 0 0];
> > stride=[50 50 50] % making it symmetrical b/c I'm still figuring out how
> > HDF5 flips
> > % dimensions
> > count=[]; block=[]; % "If count and block are specified as [], the count
> > and block
> > % size default to a single element."
> > H5S.select(space,'H5S_SELECT_SET',start,stride,count,block);
> >
> > ??? Error using ==> hdf5lib2
> > The HDF5 library encountered an error: "unable to set hyperslab
> > selection"
> >
> > Error in ==> H5S.select_hyperslab at 35
> > H5ML.hdf5lib2('H5Sselect_hyperslab', spaceID, op, start, stride, count,
> > block);
> >
> > I am running 7.8.0 R2009a.
> >
> > Thanks again,
> > Tess
>
> 1.
> As I understand it, chunking is defined on the internals of the file, i.e
> how the data is stored/organized internally in the file. This appears to
> be something required to enable 'extendible' data sets.
> http://www.zib.de/zibdoc/zib/mpp/prog/hdf5/Tutor/extend.html
>
> A hyperslab is what you appear to think of as a 'chunk of data'. It is a
> subset of the data. I dont think the chunking of the file matters when you
> are reading a hyperslab.
>
> I am sure there is more info somewhere in the HDF documentation.
>
> 2. Using [] results in the arguments defaulting to a single element. Thats
> not what you want if you want to select a single element (look at code
> below)
>
> 3. You need to use H5S.select_hyperslab
>
> 4. You need to specify a memory space ID which matches the size of the
> data selected (see code)
>
>
>
> Heres something which might help you understand better and get started
> with your own code:
>
>
>
> %hdf5write('o.h5','/DS1',reshape(1:53*512*512,[53 512 512])); %used to
> create sample data, each element ought to be unique
>
>
> fileID=H5F.open('o.h5','H5F_ACC_RDONLY','H5P_DEFAULT');
> dset=H5D.open(fileID,'/DS1'); % this is a 53x512x512 data set
> space=H5D.get_space(dset);
>
>
> start=[7 18 26]; %zero based!
> stride=[1 1 1];
> count=[1 1 1]; block=[ ];
>
>
> H5S.select_hyperslab(space,'H5S_SELECT_SET',fliplr(start-1),... %-1 to
> take care of 0 based indexing
> fliplr(stride),fliplr(count),fliplr(block)); %fliplr
> to take care of row/column major reversal (please see the ReadMe.txt in my
> previous post)
> mem_space = H5S.create_simple(3, fliplr(count), []); %see Readme.txt for
> explanation on need for flipping
>
>
> rdata = H5D.read(dset,'H5T_NATIVE_INT',mem_space,space,'H5P_DEFAULT');
>
> H5D.close(dset);
> H5S.close(space);
> H5S.close(mem_space);
> H5F.close(fileID);
>
> disp(rdata)
>
>
> %Cross check with the expected element:
> r=hdf5read('o.h5','/DS1');
> r(7,18,26)
>
>

On Tue, 03 Nov 2009 15:55:17 -0500, Tess <tess.brandon@noaa.gov> wrote:

> I think you hit on it with #4 -- I didn't know how to set the memory
> space.
>
> Apologies for accidentally leaving out the "_hyperslab" in
> "H5S.select_hyperslab" when copying from my workspace.
>
> Just to be clear, I already wrote these files to be chunked -- that's
> not the issue. The issue is whether a file needs to be written with
> hyperslabs in order to use H5S.select_hyperslab, etc. to read
> hyperslabs? Should I be using H5S.select_elements instead if I wrote
> the dataset to the file in one piece originally?

I am curious to know why you enabled chunking when you were writing?

I hope you got my point earlier that chunking and hyperslabs are not the
same. Chunking has to do with how HDF5 library internally organizes parts
of the dataset. Hyperslab concept is at the application level, it is
independent of the chunking in the file.

Have a quick look at '4.5 Storage strategies' here
http://www.hdfgroup.org/HDF5/doc/UG/UG_frame10Datasets.html

You can use any of he selection/read methods (hyperslab/elements) on a
HDF5 file irrespective of how it was written/chunked.

On Tue, 03 Nov 2009 16:09:01 -0500, Tess <tess.brandon@noaa.gov> wrote:

> I just ran your code and realized that while you defined the memory
> space to be rank 3, [x y z], the output is only one value. The way
> you've defined start, stride and count, you should have ended up with a
> 1x3 vector. I need to be able to subset, say, a [1x512x512] slice of
> data.

With this:
start=[7 18 26];
stride=[1 1 1];
count=[1 1 1]; block=[ ];

Count is the number of data points along that specific dimension.
Total number of actual data points selected is 1x1x1 = 1


Did you consider using this:

start=[1 1 1];
stride=[1 1 1];
count=[1 512 512]; block=[];

Yes! Done. Thank you.

This is a long shot, but do you have any idea whether there is any intention to add the subsetting functionality of hdfread (for HDF4) into hdf5read? That was so much simpler.

"Ashish Uthama" <first.last@mathworks.com> wrote in message <op.u2t76p0ya5ziv5@uthamaa.dhcp.mathworks.com>...
> On Tue, 03 Nov 2009 16:09:01 -0500, Tess <tess.brandon@noaa.gov> wrote:
>
> > I just ran your code and realized that while you defined the memory
> > space to be rank 3, [x y z], the output is only one value. The way
> > you've defined start, stride and count, you should have ended up with a
> > 1x3 vector. I need to be able to subset, say, a [1x512x512] slice of
> > data.
>
> With this:
> start=[7 18 26];
> stride=[1 1 1];
> count=[1 1 1]; block=[ ];
>
> Count is the number of data points along that specific dimension.
> Total number of actual data points selected is 1x1x1 = 1
>
>
> Did you consider using this:
>
> start=[1 1 1];
> stride=[1 1 1];
> count=[1 512 512]; block=[];

On Tue, 03 Nov 2009 16:48:02 -0500, Tess <tess.brandon@noaa.gov> wrote:

> Yes! Done. Thank you.
>
> This is a long shot, but do you have any idea whether there is any
> intention to add the subsetting functionality of hdfread (for HDF4) into
> hdf5read? That was so much simpler.

The team may be aware of this need, I'll pass on the message.