Simulink - Loading data on the fly

TL/DR: How do I dynamically load input data on the fly into my simulink model instead of all at once?

My simulink model requires a fairly large amount of data to be loaded, on the order of ~200 MB, almost all of which is a large 6-dimensional lookup table. For single runs, or parallel runs with only a handful of workers using parfor(), this is fine, the sims run to completion. But if I attempt to use anything close to the 64 workers available to me on my server, the sims all crash. The error message isn't very descriptive (it just says lost connection), which I assume is due to running out of memory.

I'm currently loading the big lookup table into a workspace variable and then using a From Workspace block. Even though I will traverse a significant fraction of the lookup table over the course of a simulation, at any particular moment in time in the simulation, I don't need the entire table, I'm only interpolating around a very small region for my current operating point. I'm wondering if instead of loading the entire ~200 MB lookup table at the start, if I could break up the table into many smaller pieces (say, 1 MB each) and then dynamically determine which file to load based on some ID/mapping system. Since I need to interpolate to my current operating point, I'm thinking I will need to load the neighboring indices on all dimensions in order to do the interpolation.

Is there a recommended way to load data on the fly like this? Is it possible to do it with Simulink blocks? Or would I need to use embedded matlab functions and persistent memory and pass the data from matlab back to simulink? And if so, would that have a hit on performance just due to the fact that it's using embedded matlab instead of simulink?

Thanks

[Walter Roberson]

I'm currently loading the big lookup table into a workspace variable and then using a From Workspace block.

Using "From Workspace" implies that the data is in time series format, but interpolating over portions of the data implies that it is instead a block of data rather than being in time series format.

... Unless, that is, you are creating the data matrix as a single row, and adding in a dummy time of 0 at the beginning of it, and treating the columns as a "signal" (that you might potentially reshape to get into rectangular form) ???

If you have blocks of data then you would normally create the data as Parameter blocks and set_param() before calling sim()

[Paul]

I too am curious about the use of the From Workspace block. I suppose if the last dimension of the 6D table corresponds to time, then one could use the From Workspace block to interpolate in the time dimension and output a 5D signal. But where does that 5D signal go?

It sure sounds like the 6D table should be a block parameter, probaby for the n-D Lookup Table block. Whether or not that would help with the speculated memory issues is a separate question, but I suspect it won't hurt.

[pritam]

To dynamically load data into your Simulink model, use blocks like From File or Playback, which stream data during simulation rather than loading it all upfront. For large lookup tables, consider custom code with memmapfile or split the table into chunks. For parallel runs, assign only the needed data per simulation using Simulink.SimulationInput