sharedmatrix

SHAREDMATRIX Allows any Matlab object to be shared between Matlab sessions (w/o using file I/O).

Description

SHAREDMATRIX Allows any Matlab object (e.g., struct, nd-cell, nd-matrix, sparse matrix) to be shared between multiple Matlab sessions without resorting to file I/O. The Matlab sessions must have access to the same shared memory resources, i.e., the processes are on the same physical system. This program uses shared memory functions specified by POSIX and in doing so avoids disk I/O for sharing. The program should work on any Linux variant but was only tested on Ubuntu.

*UPDATE: Thanks to contributor Andrew Smith, we now support Windows through the Boost InterProcess library. The windows version has not been tested by the first author.

% For example, assuming you have data X:

shmkey=12345;
sharedmatrix('clone',shmkey,X);
clear X;
spmd(8)
    X=sharedmatrix('attach',shmkey);
    % do something with X
    sharedmatrix('detach',shmkey,X);
end
sharedmatrix('free',shmkey);

For complete description please see my blog post:
http://smlv.cc.gatech.edu/2010/08/27/shared-memory-in-matlab/

Acknowledgements

Inplace Array: A Semi Pointer Package For Matlab inspired this file.

This file inspired Dm Utils (Data Mining Utils) and Semaphore.

• MATLAB release: MATLAB 7.11 (R2010b)
• Other requirements: * Linux: POSIX shared memory (sys/shm.h) required. * Windows: Boost InterProcess required. After R2010b, array access inlining no longer supported (${MATLABROOT}/extern/include/matrix.h). Hence <R2010b must use -D"COMPLIANTMODE", otherwise no flag.

[Ole]

Didn't I check these stars?

[Ole]

Just awesome!

I agree with Evan that the installation process is a bit rough for a compiler newbie (took me half a day to get this running). Among other things, I had to manually add code for the definition of a certain "mxArray_tag" from an "undocumented matlab" site (thanks to James Tursa for pointing this out!).

But once it is compiled it does a great job! If I detach from shared data after manipulating it, matlab does not crash. Perfect, big THANX!

Especially to Andrew Smith for porting this program to windows (thank you)!

Btw, I intend to use this programm for letting two matlab programs send output to the same parallel interface. This will (hopefully) be realized by a third program reading its instructions from shared memory. Looking forwart to testing this!

[Evan]

Wow... difficult to get installed due to missing libraries, missing code, but works like a charm:

Excellent for local parallel computing (parfor) when you're broadcasting a large variable. Shared Memory allows you to skip this step, thereby greatly speeding up your code.

[Evan]

Jesus Christ! Am I the only one having bad installation issues? Fucking A!

[valantis costa]

Nice Nice but
it works only between matlab of same version...
for example it does not work between matlab 2010a x64 and 2010a x32

[E Akbas]

 

[E Akbas]

Detach doesn't seem to work for this variable:

x = cell(2,1);
x{2} = [1];

A related question: I see that you deepcopy an mxArray to the shared memory by going through all the elements of the array. Is it not possible to copy the mxArray as a whole block using calls like memcpy and mxDuplicateArray?

[E Akbas]

I'm getting "Unable to detach shared memory." errors. What are potential reasons for this?

[Joshua Dillon]

Email me. jvdillon at gmail

[oscar escarcega]

So Joshua! Do u have Andrew´s email?

[oscar escarcega]

Ok, I am a Linux user too =), but i need to used your app in a windows machine =(. Do you have the contact of Andrew Smith? I could ask for some support directly!

[Joshua Dillon]

Hi Oscar. Sorry I do not. I don't even have access to a windows machine so it is unlikely that I will be able to obtain this information even in the foreseeable future. If you figure it out though, please do share it with us! :)

[oscar escarcega]

Hello Joshua! Do you have some examples of the Andrew Smith´s windows version working the server on cpp and the client on matlab? is It possible?

[Joshua Dillon]

Hey Dan--thanks for the feedback and letting us know about this trick--this is very useful! (I believe this is also confirmed by Przemyslaw's observations.)

On the next update I will add this example to the comments.

Thanks again for the feedback Dan and everyone else. Hopefully we can keep improving shared memory support!

Speaking of which--to any willing to take on the task... There should be a way to make sharedmatrix work better by having it follow and update the reference-to list in the mxArray data structure. This would make sharedmatrix variables behave more like "regular" variables. This is probably the currently most useful feature enhancement. Please email me if you're interested in helping!

[dan]

Joshua, thank you for this routine. It's a godsend.

When writing to a shared variable inside a parfor loop I found I couldn't use shared_x=y I had to use shared_x(1:end)=y(1:end). Like this:

sharedmatrix('clone', x_key, x); clear x;
parfor 1:10
local_x = sharedmatrix('attach', x_key);
local_y = rand(size(local_x));

% THIS DOES NOT WORK
local_x = local_y;

% but this does
local_x(1:end) = local_y(1:end);

end; %end parfor

I assume this has to do with the mysteries of MATLAB copy-on-write. Anyway just wanted to post this important tip.

ps. This is Matlab r2010b on Linux x64.

[Joshua Dillon]

Hi Przemyslaw. You are exactly correct in your observation and I'm surprised someone didn't complain about this sooner! :) In fact, this issue affected me when I was working with a 16gb cell array.

I designed it this way because I *somehow* didn't think of this simple and elegant solution! (I think because I was so focused on avoiding disk usage.) What I will do for simplicity and generality is have a killclone directive which dumps the data to disk, frees it, and loads it into shm.

I will play with alloc idea too. I avoided this orginally because I was concerned that there are too many matlab internals to support--however I could at lease provide a two-dimensional double container.

I will commit these changes as soon as I have a free moment! (Which could be a >month as I am trying to graduate.)

I do want to note that for the meantime, the current program is still acceptable. Letting the system do the swapping is effectively the same, just that it will involve one extra read from disk that isn't necessary.

I was able to load a 16gb cell into a 24gb machine (sharedmatrix would need 32gb) by letting it thrash for 20min or so. I then left the matrix in memory for the next two weeks!.

[Przemyslaw]

Hi Joshua. I really appreciate Your piece of code. I plan to use it in near future but i see one prospective drawback that could be potentially simply to overcome (i hope at least).
I mean if you want to work on really large amount of data that there is a moment when you need to store in memory two copies of it.

C = rand(13000,13000);
shmkey=1234567;
shmsiz=sharedmatrix('clone',shmkey,C);% double memory usage
clear C; %until this moment

Is it possible to add a directive like 'alloc' or something in this way to allocate a matrix in a SHM?
To fill it with data from file (f.e) so one wouldn't have two copies of the data - so swapping wouldn't be necessary.
so it could work like this

shmkey=1234567;
[C,shmsiz]=sharedmatrix('alloc',shmkey,[13000 13000],'double');
C(:,:) = rand(13000,13000);

[Joshua Dillon]

Hi Guy. As far as either of us could tell, Windows does not natively have shared memory. As I am not a Windows user though, I could be wrong.

I had hoped that we could design a platform independent version, however even within Boost the shared memory API is Windows specific.

[guy katz]

I have a question:
when And Andrew implemented the SharedMemory function why he use boost and did not use the shared memory function of windows?

[guy katz]

 

[Binlin Wu]

Josh, thank you for your emails. The code is working now for vectors. It's not critical if it works for scalar or not.

This is a great program and will be very helpful.

[Joshua Dillon]

Submitted new version. Turns out you cannot share a scalar (Alen Wu) and I committed the "open_or_create" change (Guy Katz).

[Joshua Dillon]

Thanks Guy for your feedback and correction--I will make the change and upload the fix.

[guy katz]

I am sorry my mistake there is nothing wrong with the code. It is work fine.
Although there is one thing in the part of the "clone" change from
pSegment = new windows_shared_memory(create_only, segment_name, read_write, sm_size);
to
pSegment = new windows_shared_memory(open_or_create, segment_name, read_write, sm_size);
and it will be more easy to work with the code.
tanhx

[guy katz]

I use the windows version of sharedmatrix
the SharedMemory. It seems that SharedMemory still crash when I try to attach multidimensional array.
Can some one help me.
I use the last version of SharedMemory.
Any way it is still grate function.
thank's ahead

[Joshua Dillon]

Just a heads up--I introduced a regression bug which is now solved. It should only affect sparse matrices, and probably only a cell array of sparse matrices at that.

[Joshua Dillon]

Dan--thanks for exploring this. Understanding memory statistics in *nix is indeed very difficult and I am certainly no expert! Be assured though, that the object is being shared and not copied despite top's reporting (writing to a shared matrix also verifies this).

I have uploaded a version which incorporates Andrew Smith's code for structs (should be available on Apr. 11). I also made several bug fixes and code cleaning. I hope this has not introduced any new bugs--unfortunately I have not had time to do extensive testing so please let me know if something doesn't work right!

I did not update the help info. This version really does support every Matlab object! Note however, that write support doesn't always work on cells or structs--only a simple matrix has guaranteed read/write support between processes. Every other object is only guaranteed to have read support.

[dan]

Further tests reveal... yes, top includes shared memory in its report of each process's RES and VIRT usage.

Joshua this may be worth a mention in the documentation.

[dan]

Another VERY grateful user; thank you for this contribution!

But in my tests and monitoring using top it seems that each parfor process creates a local copy of the shared global variable. Does 'top' include shared memory when reporting each process' RAM use?

Here's my test:

matlabpool('open');
C = rand(13000,13000);
shmkey=1234567;
shmsiz=sharedmatrix('clone',shmkey,C);
clear C;

parfor i = 1:5, MyTest(shmkey, i); end;

sharedmatrix('free',shmkey);

And MyTest.m():

function MyTest( shmkey, threadID )

pC = sharedmatrix('attach',shmkey);

%% This operation should not require a local copy of pC since it should be read-only.
result = nnz(pC);
result = result * threadID;
whos('pC'); % reports 1.3gb
disp(result);

sharedmatrix('detach',shmkey,pC);
clear pC;

end

When I run this test and monitor it using top, it shows each worker MATLAB process momentarily using >1gb of RAM.

Is this an artifact of 'top' -- does top report process memory inclusive of shared memory it's accessing? Or does nnz(x) actually create a local copy of x in each worker?

[Joshua Dillon]

Thanks to the awesome efforts of my new colleague Andrew Smith, we now support structs and multidimensional objects. Through Boost::Interprocess it is also platform independent! Thanks Andrew!

Stay tuned for the new release!

[Min]

As mentioned in previous posts and the included documentation, carefully detaching and free'ing the sharememory keys is essential because any slight sloppiness with managing your shared memory will absolutely punish you and crash Matlab most ungracefully.

That said, this function rocks! My previous multi-core version of my Matlab script shared very large 2D complex arrays by writing and reading ".mat" files. Sure it worked, but I could hear my disk RAID totally getting hammered and a good amount of time spent with file I/O. With the new version, which shares much (but not all) of the information via the sharedmatrix function, I saw my memory usage shrank to about half as much. Pretty cool, really.

Again, there are some particular limitations in its implementation. For instance, attaching a key to something other than a single variable, such as an array of structs or a cell array caused Matlab to crash while trying to detach the key. Even so, I found the function incredibly useful.

[Will]

This function fills an important gap in Matlab's parallel toolbox. It's common to have a large (usually read-only) dataset on which you would like to do some operations in parallel, but you would like to avoid 2 bottlenecks:

1) Having to read the data off disk (slow).

2) Having to store the same local copy on each worker (waste of memory).

I had an application which was severely RAM limited and could only use half of the cores on my local cluster, due to the need to keep common data files in memory on each worker. This function solved that problem by allowing each worker access to the same shared copy of the data. It works well and is fast.

Some implementation issues which might save you time:

1) You will need to make the changes Joshua mentioned above to sharedmatrix.c.

2) On OSX, the default maximum of shared memory is set to 4MB at boot time. You can easily increase this by creating a config file. See this page for instructions: http://www.spy-hill.com/help/apple/SharedMemory.html

3) Be careful not to clear attached shared variables in your code. As noted in the documentation, this will cause an error.

[Joshua Dillon]

As of Matlab 2010b, Mathworks has apparently dropped support of inline access to the mxArray--a feature fundamental to the functionality of this mex file. (Thanks Will for alerting me to this problem.)

The following code snippet should fix 2010b compilation errors; paste it just before the line that reads "typedef struct mxArray_tag mxArrayHack;" in sharedmatrix.c.

struct mxArray_tag {
void *reserved;
int reserved1[2];
void *reserved2;
size_t number_of_dims;
unsigned int reserved3;
struct {
unsigned int flag0 : 1;
unsigned int flag1 : 1;
unsigned int flag2 : 1;
unsigned int flag3 : 1;
unsigned int flag4 : 1;
unsigned int flag5 : 1;
unsigned int flag6 : 1;
unsigned int flag7 : 1;
unsigned int flag7a: 1;
unsigned int flag8 : 1;
unsigned int flag9 : 1;
unsigned int flag10 : 1;
unsigned int flag11 : 4;
unsigned int flag12 : 8;
unsigned int flag13 : 8;
} flags;
size_t reserved4[2];
union {
struct {
void *pdata;
void *pimag_data;
void *reserved5;
size_t reserved6[3];
} number_array;
} data;
};

[Will]

Hi Joshua,

I'm trying to compile this on OSX 10.6 (Snow Leopard) with R2010b. gcc complains about the following error:

"error: dereferencing pointer to incomplete type"

in reference to

((mxArrayHack*)*A)->data.number_array.pdata = pr;

Any idea what might be causing this? I'm not a C programmer by any stretch. Thanks in advance if you are able to come up with anything.

[Min]

 

[Joshua Dillon]

Yes, you are correct. I think that cygwin will be needed to provide shared memory functionality.

It would be possible to port this to boost::interprocess, but we should first try to compile this work using Cygwin as this keeps the code the same.

[Sebastiaan]

No idea, but that should be more about (system) libraries than compiler, right? I am sorry, but I do not have a windows version of Matlab, nor do I have windows.

[Joshua Dillon]

Hi Sebastiaan. Does LCC support shared memory operations? Are you running windows and if so, were you able to use this program?

[Sebastiaan]

For Windows, do you need Cygwin? Matlab comes with the LCC compiler in Windows. Pretty bare-bone, but a lot of people will be happy if it compiles with LCC I guess.

[Joshua Dillon]

Hi Alexander.
1) Yes! The data exists outside of Matlab hence you don't even need Matlab to access it. And of course it never uses the disk (unless the system swaps it out of course).
2) Let's work together to try to compile this with cygwin. I only have Linux but I looked into it and I think its possible. You can find my email on my website, under contact.

Cheers,
Josh

[Alexander]

Thanks for your effort.
1. Is the shared matrix stored for subsequent MATLAB sessions (when MATLAB quits and restarts).
2. If yes, I would highly appreciate having a Windows version.

[Joshua Dillon]

New version submitted; added ipcs interface whosshared.m.

[Joshua Dillon]

New version submitted; corrects mishandling of sparse logical matrices.

[Joshua Dillon]

If there is interest for a Windows version, I can easily port this code to use boost::interprocess. This would make it platform independent. Please respond if this is something you need.

[Joshua Dillon]

Thanks for the tip, although I doubt it will be that helpful. The problem is that my program necessarily bypasses the Mex API and directly manipulates the mxArray data members. To do this kind of manipulation for the Matlab "struct" object requires reverse engineering its mxArray specification. Unfortunately this is not specified by matrix.h (cf. "struct mxArray_tag"). Although I do have an idea how to do this, it requires hard-coding a new C-struct, rather than just typedef'ing the one specified by matrix.h which decreases the chances that this program will work on older versions of Matlab.

I will still look into your suggestion though. I appreciate the help!

[Sebastiaan]

Thanks Joshua. The forced re-read I did is a result of array manipulation outside Matlab's scope I use myself (e.g. call Twice(A) to double all elements of A without copying memory, but Matlab does not know then that A has changed, so it does not update pointers to the same data).

Maybe this will give you ideas on how to copy structures:
http://www.mathworks.com/matlabcentral/fileexchange/25656-compression-routines

This package has routines which copies any Matlab structure (except sparse matrices) to a byte array, and reconstructs it back.

[Joshua Dillon]

By the way, Sebastiaan, you might be able to assign a struct member a shared matrix. If the copy-on-write mechanism works the way I hope it will, then the shared memory linkages this tool constructs *should* be preserved.

Having said that though, it is just as likely that it won't work. I considered supporting structs but cell arrays were hard enough and believe this covers most people's needs (or they could adapt their code if necessary). I will however put this on the TODO list for version 2.

One more issue of confusion I just realized from your second question: you do not need to force Matlab to reread memory. The shared object points to exactly the same piece of memory in your system. When you change one matrix you are not "updating" the other matrix--it is literally the same data.

However, you can do local "manipulations," by design. For example, the parent can have matrix A shared and the child can be A' (transposed) yet they will still have the same shared data. The only part that is truly shared is the data itself. This was done to maximize the "sharing potential," if that makes sense.

[Joshua Dillon]

Hi Sebastiaan. Thanks for the corrections to the description--it has been updated.

The matrix that is cloned is not shared. You should clear the original to avoid confusion. To rework your example:

Parent:
A = [1 1];
shmkey=1;
sharedmatrix('clone', shmkey, A);
% at this point A is a totally local copy
clear A; % no need to have the original in memory
A = sharedmatrix('attach',shmkey); % now A exists as shared
% do something on client
% A is now [2 1]

Client:
shmkey = 1;
A = sharedmatrix('attach', shmkey);
A(1) = 2;

Thanks again for the question and correction!

[Sebastiaan]

Very nice tool to share between different Matlab sessions. I have no use for it now since I only work with structures, but I will keep this in mind.

1) There is a bug in your description on the FEX:
X=sharedmatrix('detach',shmkey);
should be:
sharedmatrix('detach',shmkey, X);

2) I do not fully understand how it works. If I share a matrix, and change a value in the attached session, the matrix does not get updated in the parent session (also when forcing Matlab to update the matrix):
Parent:
A = [1 1];
sharedmatrix('clone', 1, A);

Client:
A = sharedmatrix('attach', 1);
A(1) = 2;

Parent, force Matlab to re-read memory:
A(2) = 2;
disp(A)
[1 2]

Why is A not updated in the parent session? It is in other client sessions.

[Joshua Dillon]

Fixed version is now available.

[Joshua Dillon]

Note: this first version mishandles empty cell array members and has a typo in install_shared_matrix.m. The fix is uploaded and pending review, but can also be obtained from the blog description (see above for link).