%SHAREDMATRIX Allows 2D (cell) matrix to be shared between Matlab processes.
% SHAREDMATRIX allows certain Matlab objects (see below) to be shared
% between multiple Matlab sessions, provided they have access to the same
% shared memory resources, i.e., the processes are on the same physical
% system. This program uses shared memory functions specified by POSIX and
% will not work in Windows nor any other non-POSIX environment, although it
% probably could be compiled under cygwin.
%
% WARNING: This program manipulates Matlab objects in a manner that is
% highly unstable. It is very likely that using this function will cause
% Matlab to crash and could potentially lead to data corruption or loss.
% This instability should be limited to the Matlab process itself and
% should not affect the stability of the system.
%
% Since this program is more complicated than the typical Matlab function
% please read this manual thoroughly! It is organized in a Q/A style to
% make referring to it easier.
%
% --- What exactly can be shared?
%
% This program allows multiple Matlab sessions to access one copy of the
% following Matlab data objects:
% - 2D matrix, non-sparse, non-empty
% - 2D matrix, sparse, non-empty
% - 2D cell array of above and/or empty matrices (at least one non-empty)
% - 2D cell array of any of above (recursive)
% In principle, the matrices can be of any type, i.e., UINT32, UINT64,
% etc., although only the DOUBLE type has been extensively tested. The
% matrices can be real or complex.
%
% --- What is meant by "shared?"
%
% A "shared object" is a Matlab matrix or cell array with references to a
% segment of "shared memory." Shared memory is special in that it can be
% made accessible to any program. For purposes of this program the shared
% data object should generally be regarded as read-only, however, by design
% there is no mechanism to force this.
%
% Writing to shared objects can have unpredictable results for two reasons:
% - If two or more sessions write to the same element at approximately
% the same time, the result is non-deterministic. This phenomenon is
% called "resource contention."
% - This program violates standard Matlab data reference conventions.
% Practically speaking this means that the copy-on-write mechanism
% will cause shared objects to become unshared or partially unshared
% or simply crash Matlab.
%
% If you do write to shared memory it is recommended that you share a
% non-sparse matrix and write only to internal elements, i.e., no dynamic
% resizing. Writes to a shared sparse matrix should not result in a change
% to nzmax. As the object becomes more complicated, i.e., sparse matrices
% followed by cell arrays, Matlab cross-referencing becomes more
% complicated and it becomes harder to trick Matlab into using shared
% memory.
%
% --- What are the typical use-cases for this program?
%
% Typically, this program is most useful for read-only sharing of a large
% matrix among several local Matlab worker sessions, perhaps in the body of
% a PARFOR or SPMD. This situation occurs, for example, when fitting a
% model to a large amount of training data or evaluating a model under
% different parametrizations.
%
% This program is also useful when repeatedly loading a large amount of data,
% i.e., from a .mat file. One could prevent this by loading the data into
% shared memory and attaching it as needed rather than reloading it. Since
% shared memory persists unless explicitly freed, it can easily be reattached
% to Matlab even if Matlab crashes or is exited. In this way there is
% essentially zero time to load large data.
%
% --- Enough already! How do I use this thing?!
%
% Suppose you have two running Matlab sessions, S0 and S1, and a large sparse
% matrix, X, which is loaded on S0.
%
% % S0:
% shmkey = 12345;
% shmsiz = sharedmatrix('clone',shmkey,X);
% clear X; % not required but no need to keep X locally
% % wait for S1 to finish, perhaps S1 is a spmd worker
% sharedmatrix('free',shmkey);
%
% % S1:
% shmkey = 12345; % must match the key!
% X = sharedmatrix('attach',shmkey);
% % do something with X
% sharedmatrix('detach',shmkey,X);
% clear X; % not required but good practice
%
% --- What are "directives?"
%
% This program has four modes of operation which are referred to as
% "directives." Directives indicate how to manipulate the memory, as
% follows:
% 'clone' copy data to shared memory
% 'attach' reconstitute shared memory as a local object
% 'detach' destroy the local object
% 'free' mark the shared memory segment for destruction
% All directives minimally require an integer value, or key, which uniquely
% identifies a particular shared memory segment.
%
% --- How do I copy a variable to shared memory?
%
% The "clone" directive copies the unshared Matlab object into shared
% memory. When in shared memory, the object is stored in a custom format
% optimized to use the least amount of memory possible.
%
% Required Arguments:
% (1) a unique integer "key" to identify a shared memory segment.
% (2) the variable to copy.
% Returns:
% (1) the size of the shared memory segment (bytes).
%
% --- How do I load shared memory into Matlab?
%
% The "attach" directive creates a Matlab object that is a shallow copy of
% shared memory data. Locally this object "owns" very little data as most
% is a reference to shared memory. The attached shared memory object is
% essentially a reference to data stored in shared memory.
%
% Required Arguments:
% (1) a unique integer "key" to identify a shared memory segment.
% Returns:
% (1) the attached shared memory object.
%
% --- How do I remove shared memory from Matlab?
%
% The "detach" directive removes the references the attached shared memory
% object makes to shared memory and replaces them with dummy data. This
% prevents the Matlab garbage collector from discovering any oddities.
%
% Required Arguments:
% (1) a unique integer "key" to identify a shared memory segment.
% (2) the variable to detach.
% Returns:
% (nil)
%
% --- How do I free shared memory from my system?
%
% The "free" directive marks the shared memory segment for deletion. Note:
% it is not actually deleted until every attached session explicitly
% detaches or is terminated. As soon as the last session detaches, the
% system will reclaim the allocated segment. The contact passphrase is
% ABEND, see second and third to last questions.
%
% Required Arguments:
% (1) a unique integer "key" to identify a shared memory segment
% Returns:
% (nil)
%
% --- How do I prevent crashes?
%
% Matlab will crash when its internal garbage collection is run on an
% attached shared object. This means that you cannot do the following:
% - CLEAR an attached shared object
% - use PACK when attached shared objects exist.
% In some circumstances, you may not be able to save a shared object,
% although this statement has not been thoroughly tested.
%
% The best way to prevent crashes is to attach the shared object only as it
% is needed and ALWAYS use the detach directive when done. Continually
% reattaching the same object without detaching will result in incorrect
% memory reporting and may cause the system to unnecessarily allocate
% additional resources, i.e., "OOM kill" other processes (out-of-memory).
%
% --- How do I manage shared memory outside of this program?
%
% Often this program will be insufficient for working with shared memory
% objects. A common situation is that Matlab will crash before memory can be
% appropriately detached or freed. Linux provides several commands which
% should augment this program.
%
% Useful commands/tips/tricks for managing shared memory in Linux:
% 1) To see current shared memory maximum (bytes):
% cat /proc/sys/kernel/shmmax
% cat /proc/sys/kernel/shmall
% 2) To do a one-time change shared memory maximum (bytes):
% sudo sysctl -w kernel.shmmax=25323843584 # 1TB
% sudo sysctl -w kernel.shmall=6182579 # 1TB/(pagesize=4096)
% 3) To permanently change the shared memory maximum (bytes):
% Edit /etc/sysctl.conf, i.e.,
% sudo vim /etc/sysctl.conf
% and add the following three lines to the end of the file,
% # default shared memory maximum
% #kernel.shmmax = 33554432
% kernel.shmmax = 25323843584 # 1TB
% kernel.shmall = 6182579 # 1TB/(pagesize=4096)
% then load the changes via,
% sudo sysctl -p /etc/sysctl.conf
% 4) To see the currently open shared memory resources:
% watch --interval=1 ipcs -m # or just: ipcs -m
% 5) To delete (destroy) a shared memory resource:
% ipcrm shm xxxxxx
% 6) To delete (destroy) all shared memory for current user:
% for id in `ipcs -m|grep "$USER"|cut -c12-19`;do ipcrm shm $id; done
%
% --- I found a bug, what can I do?
%
% First: calm self. This is a pretty wacky program so you shouldn't be
% surprised when you encounter a problem.
% Second: simplify the problem into a 10 or maybe 20 line .m script and zip
% it up with the necessary .mat data files and email it to the author
% at: "jvdillon {at} gmail {dot} com" with a description of the fault.
% I will not respond to any email that doesn't have the secret word in
% the subject, which you can find from reading this manual.
%
% This program was only tested on Matlab 7.8.0.347 (R2009a). It is
% *certain* that it will not work on older versions. Send me your
% $(MATLABROOT)/extern/include/matrix.h
% and I will try to make the necessary additions to this program.
%
% --- I looked at your code and it sucks!
%
% Well that isn't a question...but I hear ya. I am not a software engineer
% but I did badly need shared memory in Matlab (see "typical use-cases"
% #1). However, I would very much like to improve this program, so PLEASE
% share your complaints, criticisms, and/or code-fu. You can email me at:
% "jvdillon {at} gmail {dot} com" and I will reply as soon as I can. I
% will not respond to any email that doesn't have the secret word in the
% subject, which you can find from reading this manual.
%
% --- What's next?
%
% If this is useful to people I will write a semaphore interface for
% improved inter-Matlab communication.
%
%
%
% See also WHOSSHARED, PARFOR, SPMD, LOAD, PACK.
%
% Copyright (c) 2010,2011 Joshua V Dillon
% All rights reserved. (See file header for details.)
% Copyright 2010,2011 Joshua V Dillon
% $Revision: 0.9.0.0 $ $Date: 2010/08/27 9:24 $
% $Revision: 0.9.1.0 $ $Date: 2011/05/26 9:28 $
% Built-in function.
