Working with Codistributed Arrays
How MATLAB Software Distributes Arrays
When you distribute an array to a number of workers, MATLAB® software partitions the array into segments and assigns one segment of the array to each worker. You can partition a two-dimensional array horizontally, assigning columns of the original array to the different workers, or vertically, by assigning rows. An array with N dimensions can be partitioned along any of its N dimensions. You choose which dimension of the array is to be partitioned by specifying it in the array constructor command.
For example, to distribute an 80-by-1000 array to four workers, you can partition it either by columns, giving each worker an 80-by-250 segment, or by rows, with each worker getting a 20-by-1000 segment. If the array dimension does not divide evenly over the number of workers, MATLAB partitions it as evenly as possible.
The following example creates an 80-by-1000 replicated array and assigns it to
A. In doing so, each worker creates an identical array
in its own workspace and assigns it to variable
A is local to that worker. The second command distributes
A, creating a single 80-by-1000 array
that spans all four workers. Worker 1 stores columns 1 through 250, worker 2 stores
columns 251 through 500, and so on. The default distribution is by the last
nonsingleton dimension, thus, columns in this case of a 2-dimensional array.
spmd A = zeros(80, 1000); D = codistributed(A) end Worker 1: This worker stores D(:,1:250). Worker 2: This worker stores D(:,251:500). Worker 3: This worker stores D(:,501:750). Worker 4: This worker stores D(:,751:1000).
Each worker has access to all segments of the array. Access to the local segment is faster than to a remote segment, because the latter requires sending and receiving data between workers and thus takes more time.
How MATLAB Displays a Codistributed Array
For each worker, the MATLAB Parallel Command Window displays information about the codistributed array, the local portion, and the codistributor. For example, an 8-by-8 identity matrix codistributed among four workers, with two columns on each worker, displays like this:
>> spmd II = eye(8,"codistributed") end Worker 1: This worker stores II(:,1:2). LocalPart: [8x2 double] Codistributor: [1x1 codistributor1d] Worker 2: This worker stores II(:,3:4). LocalPart: [8x2 double] Codistributor: [1x1 codistributor1d] Worker 3: This worker stores II(:,5:6). LocalPart: [8x2 double] Codistributor: [1x1 codistributor1d] Worker 4: This worker stores II(:,7:8). LocalPart: [8x2 double] Codistributor: [1x1 codistributor1d]
To see the actual data in the local segment of the array, use the
How Much Is Distributed to Each Worker
In distributing an array of
N rows, if
is evenly divisible by the number of workers, MATLAB stores the same number of rows (
each worker. When this number is not evenly divisible by the number of workers,
MATLAB partitions the array as evenly as possible.
MATLAB provides codistributor object properties called
Partition that you
can use to determine the exact distribution of an array. See Indexing into a Codistributed Array for more information on
indexing with codistributed arrays.
Distribution of Other Data Types
You can distribute arrays of any MATLAB built-in data type, and also numeric arrays that are complex or sparse, but not arrays of function handles or object types.
Creating a Codistributed Array
You can create a codistributed array in any of the following ways:
Partitioning a Larger Array — Start with a large array that is replicated on all workers, and partition it so that the pieces are distributed across the workers. This is most useful when you have sufficient memory to store the initial replicated array.
Building from Smaller Arrays — Start with smaller variant or replicated arrays stored on each worker, and combine them so that each array becomes a segment of a larger codistributed array. This method reduces memory requirements as it lets you build a codistributed array from smaller pieces.
Using MATLAB Constructor Functions — Use any of the MATLAB constructor functions like
zeroswith a codistributor object argument. These functions offer a quick means of constructing a codistributed array of any size in just one step.
Partitioning a Larger Array
If you have a large array already in memory that you want MATLAB to process more quickly, you can partition it into smaller
segments and distribute these segments to all of the workers using the
codistributed function. Each
worker then has an array that is a fraction the size of the original, thus
reducing the time required to access the data that is local to each
As a simple example, the following line of code creates a 4-by-8 replicated
matrix on each worker assigned to the variable
spmd, A = [11:18; 21:28; 31:38; 41:48], end A = 11 12 13 14 15 16 17 18 21 22 23 24 25 26 27 28 31 32 33 34 35 36 37 38 41 42 43 44 45 46 47 48
The next line uses the
codistributed function to construct
a single 4-by-8 matrix
D that is distributed along the second
dimension of the array:
spmd D = codistributed(A); getLocalPart(D) end 1: Local Part | 2: Local Part | 3: Local Part | 4: Local Part 11 12 | 13 14 | 15 16 | 17 18 21 22 | 23 24 | 25 26 | 27 28 31 32 | 33 34 | 35 36 | 37 38 41 42 | 43 44 | 45 46 | 47 48
D are the same size
A exists in its full size on each worker,
while only a segment of array
D exists on each worker.
spmd, size(A), size(D), end
Examining the variables in the client workspace, an array that is
codistributed among the workers inside an
spmd statement, is
a distributed array from the perspective of the client outside the
spmd statement. Variables that are not codistributed
inside the spmd are Composites in the client outside the spmd.
whos Name Size Bytes Class Attributes A 1x4 489 Composite D 4x8 256 distributed
reference page for syntax and usage information.
Building from Smaller Arrays
codistributed function is less
useful for reducing the amount of memory required to store data when you first
construct the full array in one workspace and then partition it into distributed
segments. To save on memory, you can construct the smaller pieces (local part)
on each worker first, and then use
codistributed.build to combine
them into a single array that is distributed across the workers.
This example creates a 4-by-250 variant array A on each of four workers and
codistributor to distribute these segments across
four workers, creating a 16-by-250 codistributed array. Here is the variant
spmd A = [1:250; 251:500; 501:750; 751:1000] + 250 * (spmdIndex - 1); end WORKER 1 WORKER 2 WORKER 3 1 2 ... 250 | 251 252 ... 500 | 501 502 ... 750 | etc. 251 252 ... 500 | 501 502 ... 750 | 751 752 ...1000 | etc. 501 502 ... 750 | 751 752 ...1000 | 1001 1002 ...1250 | etc. 751 752 ...1000 | 1001 1002 ...1250 | 1251 1252 ...1500 | etc. | | |
Now combine these segments into an array that is distributed by the first dimension (rows). The array is now 16-by-250, with a 4-by-250 segment residing on each worker:
spmd D = codistributed.build(A, codistributor1d(1,[4 4 4 4],[16 250])) end Worker 1: This worker stores D(1:4,:). LocalPart: [4x250 double] Codistributor: [1x1 codistributor1d] whos Name Size Bytes Class Attributes A 1x4 489 Composite D 16x250 32000 distributed
You could also use replicated arrays in the same fashion, if you wanted to
create a codistributed array whose segments were all identical to start with.
reference page for syntax and usage information.
Using MATLAB Constructor Functions
MATLAB provides several array constructor functions that you can use to
build codistributed arrays of specific values, sizes, and classes. These
functions operate in the same way as their nondistributed counterparts in the
MATLAB language, except that they distribute the resultant array across
the workers using the specified codistributor object,
Constructor Functions. The codistributed constructor functions are listed here. Use the
codist argument (created by the
codist=codistributor()) to specify over which
dimension to distribute the array. See the individual reference pages for
these functions for further syntax and usage information.
That part of a codistributed array that resides on each worker is a piece of a larger array. Each worker can work on its own segment of the common array, or it can make a copy of that segment in a variant or private array of its own. This local copy of a codistributed array segment is called a local array.
Creating Local Arrays from a Codistributed Array
getLocalPart function copies the segments of a
codistributed array to a separate variant array. This example makes a local copy
L of each segment of codistributed array
D. The size of
L shows that it
contains only the local part of
D for each worker. Suppose
you distribute an array across four workers:
spmd(4) A = [1:80; 81:160; 161:240]; D = codistributed(A); size(D) L = getLocalPart(D); size(L) end
returns on each worker:
3 80 3 20
Each worker recognizes that the codistributed array
3-by-80. However, notice that the size of the local part,
is 3-by-20 on each worker, because the 80 columns of
distributed over four workers.
Creating a Codistributed from Local Arrays
to perform the reverse operation. This function, described in Building from Smaller Arrays, combines the
local variant arrays into a single array distributed along the specified
Continuing the previous example, take the local variant arrays
L and put them together as segments to build a new
spmd codist = codistributor1d(2,[20 20 20 20],[3 80]); X = codistributed.build(L,codist); size(X) end
returns on each worker:
Obtaining information About the Array
MATLAB offers several functions that provide information on any particular array. In addition to these standard functions, there are also two functions that are useful solely with codistributed arrays.
Determining Whether an Array Is Codistributed
returns a logical
true) if the input
array is codistributed, and logical
false) otherwise. The syntax is
spmd, TF = iscodistributed(D), end
D is any MATLAB array.
Determining the Dimension of Distribution
The codistributor object determines how an array is partitioned and its
dimension of distribution. To access the codistributor of an array, use the
This returns two properties,
spmd, getCodistributor(X), end Dimension: 2 Partition: [20 20 20 20]
Dimension value of
2 means the array
X is distributed by columns (dimension 2); and the
Partition value of
[20 20 20 20] means
that twenty columns reside on each of the four workers.
To get these properties programmatically, return the output of
getCodistributor to a variable, then use dot notation to
access each property:
spmd C = getCodistributor(X); part = C.Partition dim = C.Dimension end
Other Array Functions
Other functions that provide information about standard arrays also work on codistributed arrays and use the same syntax.
Changing the Dimension of Distribution
When constructing an array, you distribute the parts of the array along one of the
array's dimensions. You can change the direction of this distribution on an existing
array using the
redistribute function with a different
Construct an 8-by-16 codistributed array
D of random values
distributed by columns on four workers:
spmd D = rand(8,16,codistributor()); size(getLocalPart(D)) end
returns on each worker:
Create a new codistributed array distributed by rows from an existing one already distributed by columns:
spmd X = redistribute(D, codistributor1d(1)); size(getLocalPart(X)) end
returns on each worker:
Restoring the Full Array
You can restore a codistributed array to its undistributed form using the
gather takes the segments of an array that reside on
different workers and combines them into a replicated array on all workers, or into
a single array on one worker.
Distribute a 4-by-10 array to four workers along the second dimension:
spmd, A = [11:20; 21:30; 31:40; 41:50], end A = 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 spmd, D = codistributed(A), end WORKER 1 WORKER 2 WORKER 3 WORKER 4 11 12 13 | 14 15 16 | 17 18 | 19 20 21 22 23 | 24 25 26 | 27 28 | 29 30 31 32 33 | 34 35 36 | 37 38 | 39 40 41 42 43 | 44 45 46 | 47 48 | 49 50 | | | spmd, size(getLocalPart(D)), end Worker 1: 4 3 Worker 2: 4 3 Worker 3: 4 2 Worker 4: 4 2
Restore the undistributed segments to the full array form by gathering the segments:
spmd, X = gather(D), end X = 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 spmd, size(X), end 4 10
Indexing into a Codistributed Array
While indexing into a nondistributed array is fairly straightforward,
codistributed arrays require additional considerations. Each dimension of a
nondistributed array is indexed within a range of 1 to the final subscript, which is
represented in MATLAB by the
end keyword. The length of any dimension can
be easily determined using either the
With codistributed arrays, these values are not so easily obtained. For example,
the second segment of an array (that which resides in the workspace of worker 2) has
a starting index that depends on the array distribution. For a 200-by-1000 array
with a default distribution by columns over four workers, the starting index on
worker 2 is 251. For a 1000-by-200 array also distributed by columns, that same
index would be 51. As for the ending index, this is not given by using the
end keyword, as
end in this case refers to
the end of the entire array; that is, the last subscript of the final segment. The
length of each segment is also not given by using the
size functions, as they only return the length of the entire
colon operator and
end keyword are two of the
basic tools for indexing into nondistributed arrays. For codistributed arrays,
MATLAB provides a version of the
colon operator, called
codistributed.colon. This actually is a function, not a
symbolic operator like
When using arrays to index into codistributed arrays, you can use only
replicated or codistributed arrays for indexing. The toolbox does not check to
ensure that the index is replicated, as that would require global
communications. Therefore, the use of unsupported variants (such as
spmdIndex) to index into codistributed arrays might create
Example: Find a Particular Element in a Codistributed Array
Suppose you have a row vector of 1 million elements, distributed among several
workers, and you want to locate its element number 225,000. That is, you want to
know what worker contains this element, and in what position in the local part
of the vector on that worker. The
provides a correlation between the local and global indexing of the
D = rand(1,1e6,"distributed"); %Distributed by columns spmd globalInd = globalIndices(D,2); pos = find(globalInd == 225e3); if ~isempty(pos) fprintf(... 'Element is in position %d on worker %d.\n', pos, spmdIndex); end end
If you run this code on a pool of four workers you get this result:
Worker 1: Element is in position 225000 on worker 1.
If you run this code on a pool of five workers you get this result:
Worker 2: Element is in position 25000 on worker 2.
Notice if you use a pool of a different size, the element ends up in a different location on a different worker, but the same code can be used to locate the element.
As an alternative to distributing by a single dimension of rows or columns, you
can distribute a matrix by blocks using
'2dbc' or two-dimensional
block-cyclic distribution. Instead of segments that comprise a number of complete
rows or columns of the matrix, the segments of the codistributed array are
2-dimensional square blocks.
For example, consider a simple 8-by-8 matrix with ascending element values. You
can create this array in an
spmd statement or communicating
spmd A = reshape(1:64, 8, 8) end
The result is the replicated array:
1 9 17 25 33 41 49 57 2 10 18 26 34 42 50 58 3 11 19 27 35 43 51 59 4 12 20 28 36 44 52 60 5 13 21 29 37 45 53 61 6 14 22 30 38 46 54 62 7 15 23 31 39 47 55 63 8 16 24 32 40 48 56 64
Suppose you want to distribute this array among four workers, with a 4-by-4 block as the local part on each worker. In this case, the worker grid is a 2-by-2 arrangement of the workers, and the block size is a square of four elements on a side (i.e., each block is a 4-by-4 square). With this information, you can define the codistributor object:
spmd DIST = codistributor2dbc([2 2], 4); end
Now you can use this codistributor object to distribute the original matrix:
spmd AA = codistributed(A, DIST) end
This distributes the array among the workers according to this scheme:
If the worker grid does not perfectly overlay the dimensions of the codistributed
array, you can still use
'2dbc' distribution, which is block
cyclic. In this case, you can imagine the worker grid being repeatedly overlaid in
both dimensions until all the original matrix elements are included.
Using the same original 8-by-8 matrix and 2-by-2 worker grid, consider a block size of 3 instead of 4, so that 3-by-3 square blocks are distributed among the workers. The code looks like this:
spmd DIST = codistributor2dbc([2 2], 3) AA = codistributed(A, DIST) end
The first “row” of the worker grid is distributed to worker 1 and worker 2, but that contains only six of the eight columns of the original matrix. Therefore, the next two columns are distributed to worker 1. This process continues until all columns in the first rows are distributed. Then a similar process applies to the rows as you proceed down the matrix, as shown in the following distribution scheme:
The diagram above shows a scheme that requires four overlays of the worker grid to accommodate the entire original matrix. The following code shows the resulting distribution of data to each of the workers.
spmd getLocalPart(AA) end
Worker 1: ans = 1 9 17 49 57 2 10 18 50 58 3 11 19 51 59 7 15 23 55 63 8 16 24 56 64 Worker 2: ans = 25 33 41 26 34 42 27 35 43 31 39 47 32 40 48 Worker 3: ans = 4 12 20 52 60 5 13 21 53 61 6 14 22 54 62 Worker 4: ans = 28 36 44 29 37 45 30 38 46
The following points are worth noting:
'2dbc'distribution might not offer any performance enhancement unless the block size is at least a few dozen. The default block size is 64.
The worker grid should be as close to a square as possible.
Not all functions that are enhanced to work on
'1d'codistributed arrays work on