|
This is moderately lengthy, but if you know anything about
or are interested in memory mapping (namely accessing the
mapped file), I ask that you read on.
I’m having some issues indexing a mapped a file using
memmapfile. I have around 1100 uniquely named variables
(mixed data types, int32, single, uint8, char, etc) and the
sequence is repeated (in this example) about 13000 times. I
have the file mapped correctly; that is, the ‘format’ field
of the call to memmapfile contains the offset, the data
types, sizes, and the unique variable names and the sequence
repeats and takes the entire file into account.
So if I wanted to see “var0001” (the first variable, which
is simply 1:13000) across all of the 13000 repeated
“frames”, I would simply like to call:
MappedData0001 = [m.Data.var0001] and have it return
MappedData0001 = [1 2 ... 12999 13000]
However, I get the error:
??? Error using ==> memmapfile.memmapfile>subsref
A subscripting operation on the Data field attempted to create a
comma-separated list. The memmapfile class does not support
the use of
comma-separated lists when subscripting.
Ok fine, so there is no vectorized access to the mapped file
(which is unfortunate), but I guess I can live with that.
To circumvent this shortcoming, I have to loop:
MappedData0001 = single(zeros(1,13000)); % pre-allocate and
ensure the proper data type, “single” in this case
for ii = 1:13000
MappedData0001(ii) = m.Data(ii).var0001;
end
However, the first time through the loop, when the memory
map is accessed for the first time, it takes quite a bit of
time (~30 seconds) and matlab’s memory usage balloons from
about 150 mb to 1.2 gig (the file I am mapping is only
around 60 mb). However, “whos” says there are only 531075
bytes in the workspace. Then it takes more time to finish
the rest of the loop. Where is all of this overhead going?
I’ve seen the other “work around” that suggests assigning
m.Data to a variable and then using vectorized access of the
structure. However, this seems ridiculous to me: When I do
this, I watch the mem usage go to 2 gigs, back down around
1, up and down, and finally completing the process after
about a minute. However, now in the workspace, there is a
900 mb structure and matlab is now gobbling up 2.1 gigs.
So it seems like there is some "invisible" 900 mb structure
in addition to the physical structure in the workspace. I
saw in another post
(http://www.mathworks.com/matlabcentral/newsreader/view_thread/171113#438218)
that “even though it doesn't consume physical memory until
each page is accessed, the entire file DOES consume virtual
memory space. That is, every byte of the file has a
corresponding address in the mapping. So an active
memmapfile can take a huge chunk of contiguous space in the
MATLAB process...”
How is this hidden 900 mb when the original binary file
being mapped is only around 60 mb? Isn’t the point of
memory mapping to load only the data of interest into memory
anyway and be able to exclude all the other data?
|
Feed for this Thread