Is there a way to reduce the amount of RAM/Disk used during parallel simulations (parsim/batchsim)? Ex: Receiving "Out of Memory" error

I am simulating a model using "parsim". When I perform a small number of simulations, the memory usage is below my hardware limits, however when I perform many more simulations in parallel, the RAM usage increases steadily after each simulation and I run out of memory. There are a lot of signals being logged by the model. 

How can I reduce the amount of RAM that my parallel simulations are using?

Answer 1

[MathWorks Support Team]

Optimizing Memory Usage in Parallel Simulations with Simulink

Each time a parallel simulation is executed, the resulting data is transmitted from the worker to the client and retained in the system's memory (RAM) to construct the 'Simulink.SimulationOutput' object. This is the final data that 'parsim' returns to the MATLAB workspace. As the number of parallel simulations increases, the memory consumption will also rise in proportion to the data generated by each simulation.

To mitigate this, consider the following strategies to minimize the memory footprint of your simulations:

1. Reduce Logged Data Volume

Decrease the quantity of logged simulation data. Consult the following documentation for methods like decimation, limiting data points, and setting logging intervals:

Limit the Size of Logged Data - MATLAB & Simulink (mathworks.com)

You can also temporarily modify these settings using the Data Logging Override feature:

Override Signal Logging Settings - MATLAB & Simulink (mathworks.com)

2. Leverage 'postSimFcn' for Data Management

Utilize the 'postSimFcn' callback to discard unnecessary data or to save it to a file system. This function is executed on the workers, allowing for efficient parallel post-processing. Learn more about setting up a 'postSimFcn':

Set MATLAB function to run after each simulation - MATLAB setPostSimFcn (mathworks.com)

%% Example of how to apply a postSimFcn to a simulation
simIn = simIn.setPostSimFcn(@(x) postProcessLargeData_removeFields(x));

2a. Post-process Data and Remove Unneeded Fields

To manage the data within 'SimulationOutput', you can either eliminate specific fields or create a new object tailored to your needs:

function simOut = postProcessLargeData_removeFields(simOut)
    % Extract and process large data
    largeData = simOut.yout.get("OUTERDATA").Values.Data;
    metrics.meanValue = mean(largeData);
    metrics.maxValue = max(largeData);
    
    % Add a new field for metrics
    simOut.metrics = metrics;
    
    % Remove unneeded fields from output
    simOut.yout = []; 
end

function newSimOut = postProcessLargeData_clearOut(simOut)
    % Extract and process large data
    largeData = simOut.yout.get("OUTERDATA").Values.Data;
    metrics.meanValue = mean(largeData);
    metrics.maxValue = max(largeData);
    
    % Create a new SimulationOutput object with only metrics
    newSimOut = Simulink.SimulationOutput();
    newSimOut.metrics = metrics;
end

2b. Store Data Externally

Transfer simulation data to an external location such as cloud storage or a network drive to save memory:

function simOut = uploadDataToCloud(simOut, bucketname)
    % Extract large data set
    largeData = simOut.yout.get("COUNTERBUS");
    
    % Save to cloud storage (e.g., AWS S3, Azure, Google Cloud)
    save("s3://" + bucketname + "/fullSignalData.mat", "largeData");
    
    % Optionally, clear the large data set to conserve memory
    simOut.yout = [];
end

2c. Utilize MATLAB File Store

File Stores in MATLAB allow for efficient data sharing between workers and the client without occupying memory. Data can be accessed sequentially from the cluster, and you can choose specific data to persist:

function simOut = saveDataToFileStore(simOut, key)
    % Extract and save large data set
    largeData = simOut.yout.get("OUTERDATA").Values.Data;
    save("myDataFile.mat", "largeData");
    
    % Transfer the file to a FileStore
    store = getCurrentFileStore();
    copyFileToStore(store, "myDataFile.mat", key);
    
    % Optionally, remove the large data set to free memory
    simOut.yout = [];
end

3. Opt for "Log Dataset to File" During Simulation

Enable the "Log Dataset to file" option to save logged signals directly to a MAT-file for each parallel simulation. The 'SimulationOutput' object will contain a 'DatasetRef' instead of the actual data, pointing to the MAT-file's location, thereby reducing memory usage:

Running Multiple Simulations - MATLAB & Simulink (mathworks.com)

By implementing these strategies, you can significantly reduce the memory demands of your parallel simulations in Simulink.