Thread Subject: querying an hdf5 file

Hi,

This is a really basic question that I just can't seem to answer: how do I query a range of data within a hdf5 file within MATLAB using low-level hdf5 functionality?

I can read hyperslabs of data with no issue, but it's all "index-based", or in other words I need to an analogue of the "find" function within matlab to grab data in a certain range *regardless* of its index location.

Simple example:

My (huge) hdf5 file is

1 10
2 5
11 6
12 8
.
..

So if I read the "hyperslab" consisting of the first two rows, then I get

1 10
2 5

But I want to be able to just grab "consecutive data", i.e. groups of

1 10
2 5

and

11 6
12 8

from this monster hdf5 file, i.e. use a "find" function.

Any help would be greatly appreciated.

Thanks,
ZO
 

A FIND like function would require the data to be in memory, there is no
way around this.

If your hdf5 file is too large to fit in memory, try
processing/searching it in blocks. For example, read 100 rows at a time
and perform your FIND. Consider writing a helper function which uses the
low level hyperslab reading functions to do this.

Ashish Uthama <first.last@mathworks.com> wrote in message <illpur$prb$1@fred.mathworks.com>...
> A FIND like function would require the data to be in memory, there is no
> way around this.
>
> If your hdf5 file is too large to fit in memory, try
> processing/searching it in blocks. For example, read 100 rows at a time
> and perform your FIND. Consider writing a helper function which uses the
> low level hyperslab reading functions to do this.

Hi Ashish,

Thanks for your answer. I think I may have to resort to doing what you suggest -- note that there is "fast query" for hdf files using compressed bitmaps but it isn't available yet. My hope was that the H5ML functions which MATLAB wrote on its own, i.e. not just writing wrappers for hdf5 functions, contained something I missed.

It's a pretty silly situation: what good is a database I can't query?

thanks,
b