Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
memmapfile data access

Subject: memmapfile data access

From: Garrick

Date: 10 Jan, 2011 20:32:04

Message: 1 of 8

I'm working with large datasets and want to start using the memmapfile class to manage this data through mapped files.

For the sake of compatibility with previous code, I'd like to be able to rename/restructure the .Data field of a memmapfile object - essentially reference all or part of the data with another variable. Say I have a memmapfile object "a" containing data "foo" and "bar" (arrays of doubles, say). I'd like to be able to do this:

S = a.Data; % S is a structure with fields foo, bar

or this:

S.foo = a.Data.foo; % S is a structure with field foo

or this:

S = a.Data.foo; % S is an array containing foo

Now, matlab allows this, the problem is that when I assign a.Data to S, matlab copies the data from a.Data into memory to create S. This has two issues: 1) S is now a separate entity - modifications to S no longer operate on the mapped file and 2) Since S is a copy of a.Data, I've eliminated my memory savings from using mapped files in the first place.

Is there a way to do this? Essentially create a new handle/reference to the memmapfile object data without actually copying the data into memory?

Subject: memmapfile data access

From: I am so much of me

Date: 10 Jan, 2011 20:41:24

Message: 2 of 8

On 11-01-10 02:32 PM, Garrick wrote:
> I'm working with large datasets and want to start using the memmapfile class
> to manage this data through mapped files.
> For the sake of compatibility with previous code, I'd like to be able to
> rename/restructure the .Data field of a memmapfile object - essentially
> reference all or part of the data with another variable. Say I have a
> memmapfile object "a" containing data "foo" and "bar" (arrays of doubles,
> say). I'd like to be able to do this:
>
> S = a.Data; % S is a structure with fields foo, bar
>
> or this:
>
> S.foo = a.Data.foo; % S is a structure with field foo
>
> or this:
>
> S = a.Data.foo; % S is an array containing foo
>
> Now, matlab allows this, the problem is that when I assign a.Data to S, matlab
> copies the data from a.Data into memory to create S. This has two issues: 1) S
> is now a separate entity - modifications to S no longer operate on the mapped
> file and 2) Since S is a copy of a.Data, I've eliminated my memory savings
> from using mapped files in the first place.
>
> Is there a way to do this? Essentially create a new handle/reference to the
> memmapfile object data without actually copying the data into memory?

Subclass from handle class, I'd guess.

Subject: memmapfile data access

From: Malcolm Lidierth

Date: 10 Jan, 2011 21:09:05

Message: 3 of 8

Will
S = a.Data.foo(1:1000)
not do the trick?

If not look at adcarray on the FEX. That creates a custom object which contains a memmapfile object.
myObj=adcarray(a...)
myObj is passed by value but the memmapfiles are passed by reference so you only have 1 copy.
Its designed specifically for a particular task but illustrates a principle that may help.

A more generic class using handle as suggested above has been developed but is not posted yet. It could be if it would help - but only 1 field is supported in the Data i.e. either foo or bar but not both as in your example.
ML

http://sigtool.sourceforge.net/

Subject: memmapfile data access

From: Garrick

Date: 10 Jan, 2011 21:17:05

Message: 4 of 8

> Subclass from handle class, I'd guess.

@so much of me - That's one option I was considering, basically make a handle subclass that creates a structure of handles to the memmapfile data. Has anyone tried something similar?

Subject: memmapfile data access

From: Garrick

Date: 10 Jan, 2011 21:54:05

Message: 5 of 8

Thanks for the reply, Malcolm.

> Will
> S = a.Data.foo(1:1000)
> not do the trick?
>

This is the crux of the issue - it does work, but, as far as I'm aware, it creates a copy of the data in S. My main concern is to avoid allocating memory in matlab, keeping as much data / data manipulation in the mapped file as possible.

> If not look at adcarray on the FEX. That creates a custom object which contains a memmapfile object.
> myObj=adcarray(a...)
> myObj is passed by value but the memmapfiles are passed by reference so you only have 1 copy.
> Its designed specifically for a particular task but illustrates a principle that may help.
>
> A more generic class using handle as suggested above has been developed but is not posted yet. It could be if it would help - but only 1 field is supported in the Data i.e. either foo or bar but not both as in your example.

I took a look at adcarray - it's an interesting tool, and conceptually sort of what I'm looking for. However, while adcarray creates and object that reads like an array (e.g you can do x = myObj(:,1), where myObj is an adcarray object), I would like to create an object that reads like a structure, referencing multiple fields in the data (e.g. I could reference myObj.foo(:,1), where foo is a reference to a memmapfile field a.Data.foo). Any ideas on how to approach that? Your generic class sounds like an interesting jumping off point, so let me know if you decide to post it. Thanks!

Subject: memmapfile data access

From: Garrick

Date: 10 Jan, 2011 22:23:05

Message: 6 of 8

A more succinct way of stating what I want:

I have a memmapfile object a. I would like to change the interface so that instead of referencing a.Data.foo, I could simply reference a.foo.

a.foo would still be a reference to a location in a mapped file, rather than a copy in matlab memory.

Subject: memmapfile data access

From: Malcolm Lidierth

Date: 11 Jan, 2011 01:47:05

Message: 7 of 8

Garrick
I think I have caught on

How about this skeleton:
--------------------------------------------------------------------
classdef myobj
    
    properties
        mymemmap=[];
    end
    
    methods
        
        function obj=myobj(memmapfileoninput)
            obj.mymemmap=memmapfileoninput;
            return
        end
        
        function varargout=subsref(obj, index)
            switch index(1).type
                case '.'
                    switch lower(index(1).subs)
                        case 'foo'
                            varargout{1}=obj.mymemmap.Data.foo(index(2).subs{:});
                        case 'bar'
                            varargout{1}=obj.mymemmap.Data.bar(index(2).subs{:});
                    end
            end
            
        end
        
    end
end
------------------------------------------------------------------------------------
You will need to add a subsasgn method also for the write access.
Note if you do not need write-enabled memmapfiles, you could forget subsref/subsasgn altogether and just create two methods called foo and bar. On the RHS, MATLAB makes no syntactical distinction between methods and properties (2nd thought: not sure if that's the case with dynamic property names - need to check).

The memmapfile is a reference so you retain 1 copy.

If you subclass myobj to hgsetget you will have immediately have get/set etc with no extra work.

>> s.Data.foo=[1:10];
>> s.Data.bar='abcdefg'

s =

    Data: [1x1 struct]

>> obj=myobj(s)

obj =

  myobj

  Properties:
    mymemmap: [1x1 struct]

  Methods

>> obj.foo(1:5)

ans =

     1 2 3 4 5
Regards
Malcolm

Subject: memmapfile data access

From: Garrick

Date: 11 Jan, 2011 15:07:03

Message: 8 of 8

Malcolm,

I think that's the approach I'm considering - building a class with subasgn / subsref methods that allow direct access to the data fields of a memory map with some creative parsing. The data I'm working with has a large number of fields and the field names aren't always identical from set to set, so using methods as stand-ins for the field names would be unwieldy.

Thanks for your help,
-Garrick

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us