How do I share data between my parallel pool workers?

I am running a script in parallel using the Parallel Computing Toolbox. As part of this, I would like to save important data from one worker such that it is readable from another worker in the same job.

Thus far, I have tried to write files containing the data to my device's disk and have the other workers read those files as necessary. However, this often fails to work as expected. Is there a better way to do this?

Answer 1

[MathWorks Support Team]

As of MATLAB R2022a, MATLAB parallel jobs automatically generate a "ValueStore" object. This object stores data values and associated keys out of system memory, providing easy access across workers in a parallel job or the opportunity to retrieve intermediate results.

This object provides many benefits regarding your desired workflow, including the following:

• Key-value pairs allow you to control exactly how your data is stored and accessed.
• The values in a "ValueStore" object are not stored in system memory, meaning they can store larger results.
• Compared to writing and reading to files, a "ValueStore" may increase execution speed and decrease the opportunity for errors to arise.

For more information on this property, as well as how to utilize it, please see the following documentation page:

https://www.mathworks.com/help/parallel-computing/parallel.valuestore.html

Additionally, should you want to save files rather than data, I recommend looking into the "FileStore" object. This object mirrors the "ValueStore", but saves files rather than individual data values. For more information, see the following documentation:

https://www.mathworks.com/help/parallel-computing/parallel.filestore.html

Note that these properties may not apply to every workflow involving data management. For instance, in some cases, a "datastore" or "parallel.pool.Constant" may provided superior utility for any given workflow. To help determine the best method to manage your data when running in parallel, reference the following documentation:

https://www.mathworks.com/help/parallel-computing/choose-how-to-manage-data-in-parallel-computing.html