How do I see what parsim/batchsim is doing during worker setup to debug parsim/batchsim worker setup issues?

I am running my project with parsim/batchsim and I am running into some issues during the setup phase on the workers. Those issues might include:

1. A variable or file is missing or cannot be found on the workers. Error messages include "file not found", "Unrecognized function or variable", "Variable 'myVariable' does not exist."
2. A variable or file has a different value than expected.
3. I want to make sure my simulation cache files are being transferred and used. 

Answer 1

[MathWorks Support Team]

parsim and batchsim will run an automated setup on the workers for simulation. In most cases, the setup should work and the simulations should run as expected. In rare cases, the automated setup may fail to initialize the workers as you expect. Common causes of setup issues include user-written callbacks causing unintended consequences when running on workers, and inlined file locations that don't exist on workers. The following guide will show how to use diaries to debug parsim/batchsim setup.  

Worker Diaries

In R2023a, worker diaries were added to the SimulationMetadata returned in the SimulationOutput object for parsim/batchsim. We can use the preSimFcn to print diagnostics to this diary to help us debug parsim and batchsim setup issues. 

You can access the diary like so:

>>out = parsim(in);
>>out(1).SimulationMetadata.ExecutionInfo.Diary

In the diary, you will see anything printed to the command window on the worker. This includes:

• Project startup/shutdown callbacks
• preSimFcn and postSimFcn callbacks
• model and block callbacks
• Model Reference rebuild messages

Prior to 23a, you can write code to create your own worker diary. See the attached example for details.

Inject Debugging Code in preSimFcn

Particularly the preSimFcn can be useful to debug setup issues. Here is some example code:

function in = preSimDiary(in)
    %print out some interesting information
    fprintf('%%%%Worker Info%%%%\n')
    disp(getCurrentWorker)
    fprintf('%%%%Worker''s Current Directory and Contents%%%% \n %s \n %s\n',pwd,ls);
    fprintf('%%%%Model File Location%%%%\n %s \n',which(in.ModelName));
    fprintf('%%%%Cache Folder Location and Contents%%%% \n') 
    cfg = Simulink.fileGenControl('getConfig');
    disp(cfg.CacheFolder)
    disp(ls(cfg.CacheFolder))
    fprintf('%%%%Attached Files Location and Contents%%%% \n %s \n',getAttachedFilesFolder('myFile.mat')) 
    disp(ls(getAttachedFilesFolder('myFile.mat')))
    fprintf('%%%%Base Workspace on Worker%%%% \n')
    evalin('base','whos')
end

Example Diary on Local Pool

Below is an example diary output with some display statements added to the code.

Of note:

• The order that the ProjectStartup, SetupFcn, preSimFcn, and postSimFcn run.
• The worker is given its own temp directory to prevent race conditions. However, not all files are transferred to this folder. Because workers share a file system with the client, workers can simply access files from their existing location to avoid excessive file copies.
• A copy of the cache files is sent to each worker. Thus models in Rapid Accelerator mode or Model References in Accelerator mode will not be rebuilt.
• Attached files are given their own folder. Note that the relative path to these files is different on the workers than on the client. Its best not to not hard code relative or absolute paths in your model. Instead, add files to the Project path or MATLAB path and refer to them by name.
• All Base Workspace variables are transferred, including those not used by the model such as SimulationInput and SimulationOutput objects from previous runs. For this reason it is recommended to use TransferBaseWorkspace variables only for small projects. See this blog post for options. 

 Hello World! This is the Project Startup callback speaking
 Hello World! This is the SetupFcn speaking
 Hello World! This is the preSimFcn speaking
 %%Worker Info%%
  Worker
                  Host: MATHWORKS
          ComputerType: WIN64
 
             ProcessId: 7856
 
 %%Worker's Current Directory and Contents%% 
  C:\Users\rspence\AppData\Local\Temp\tp7bacffba_aa91_43ed_ba9f_ddf0efc891e3 
 %%Model File Location%%
  C:\Users\rspence\OneDrive - MathWorks\Documents\MATLAB\MyTools\myExamples\parsimDiaries\ExampleToPublish\topModel.slx 
 %%Cache Folder Location and Contents%% 
 C:\Users\rspence\AppData\Local\Temp\tp7bacffba_aa91_43ed_ba9f_ddf0efc891e3          
 refModel1.slxc
 refModel2.slxc
 refModel3.slxc
 topModel.slxc 
 %%Attached Files Location and Contents%% 
  C:\Users\rspence\AppData\Local\Temp\ca19-95ea-59b5-2c4a\a\tpe3d2868a_1d6b_4bc8_9d9c_23e030e7ce18         
 myFile.mat  
 topModel.slx
 %%Base Workspace on Worker%% 
   Name                   Size            Bytes  Class                        Attributes
 
   i                      1x1                 8  double                                 
   modelName              1x8                16  char                                   
   myBaseWksVariable      1x1                 8  double                                 
   out                    1x4             45328  Simulink.SimulationOutput              
   simin                  1x4              1416  Simulink.SimulationInput                             
 ### Starting serial model reference simulation build.
 ### Model reference simulation target for refModel1 is up to date.
 ### Model reference simulation target for refModel2 is up to date.
 ### Model reference simulation target for refModel3 is up to date.
 Build Summary
 0 of 3 models built (3 models already up to date)
 Build duration: 0h 0m 2.8759s
 Hello World! This is the postSimFcn speaking

Example Diary on an External Pool

Below is an example diary when the client is Windows and the external cluster is Linux.

Of note:

• Cache files are sent to workers but cache files are platform-dependent. Thus the model needs to be rebuilt, as cache files were generated on Windows but the cluster is Linux. 
• Now the client and the workers do not share a file system so all files need to be transfered to the workers. However, all workers on the same node/machine share the same attached files location. Since these workers share the same file system, we can avoid the cost of making file copies for each worker. 

 Hello World! This is the SetupFcn speaking
 Hello World! This is the preSimFcn speaking
 %%Worker Info%%
  Worker
                  Host: MATHWORKSCLUSTER
          ComputerType: GLNXA64
                  Name: worker_4
 
 %%Worker's Current Directory and Contents%% 
  /tmp/tpc75735f8_7680_427f_b9c8_6d24ff282ff2 
 %%Model File Location%%
  /tmp/tpe6ee9323_7081_4809_b847_9c47e4088788ph3508134/project/tp9355181d_ffe1_4749_8f9b_33045b2aff76/topModel.slx 
 %%Cache Folder Location and Contents%% 
 /tmp/tpc75735f8_7680_427f_b9c8_6d24ff282ff2
 refModel1.slxc    refModel2.slxc    refModel3.slxc    topModel.slxc
 %%Attached Files Location and Contents%% 
  /var/lib/mjs/MATHWORKSCLUSTER/matlab/filedependencies/684/a/tp8136a265_5bf7_4bdf_80da_c971b2e73420 
 myFile.mat  topModel.slx
 
 %%Base Workspace on Worker%% 
   Name                   Size            Bytes  Class                        Attributes
                           
   modelName              1x8                16  char                                   
   myBaseWksVariable      1x1                 8  double                                 
   out                    1x4             46358  Simulink.SimulationOutput              
   simin                  1x4              1416  Simulink.SimulationInput                                         
 
 ### Starting serial model reference simulation build.
 ### Successfully updated the model reference simulation target for: refModel1
 ### Successfully updated the model reference simulation target for: refModel2
 ### Successfully updated the model reference simulation target for: refModel3
 
 Build Summary
 
 Simulation targets built:
 
 Model      Action                        Rebuild Reason                        
 ===============================================================================
 refModel1  Code generated and compiled.  refModel1_msf.mexa64 does not exist.  
 refModel2  Code generated and compiled.  refModel2_msf.mexa64 does not exist.  
 refModel3  Code generated and compiled.  refModel3_msf.mexa64 does not exist.  
 
 3 of 3 models built (0 models already up to date)
 Build duration: 0h 0m 36.826s
 Hello World! This is the postSimFcn speaking

Try it for Yourself

Attached is an example project that shows how you can use worker diaries to print diagnostic information.