Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Simulink: Importing Very Large Data Sets

Subject: Simulink: Importing Very Large Data Sets

From: Luke

Date: 15 Sep, 2009 04:28:01

Message: 1 of 4

I need to use very large data sets as inputs to my simulations in Simulink - some as large as ~24GB. Of course, this is not going to fit into the workspace, so I would like the equivalent of moving a file pointer through the file or some other sort of solution as I simulate data in Simulink.

My data is in a .sim file and must be interpreted as signed integers (here in M-code):
>> fid = fopen('data.sim', 'r'); %open raw data file
>> A = fread(fid, XX, 'schar'); % get XX samples of data

What I have to do right now is save as much as I can to the workspace and then use a From Workspace block, but if there is a way to input data without having to use any workspace memory, that would be greatly beneficial.

Does anybody have any suggestions?

Subject: Simulink: Importing Very Large Data Sets

From: Luke

Date: 16 Sep, 2009 04:12:02

Message: 2 of 4

"Luke " <lukeforusenet@gmail.com> wrote in message <h8n54h$6g4$1@fred.mathworks.com>...
> I need to use very large data sets as inputs to my simulations in Simulink - some as large as ~24GB. Of course, this is not going to fit into the workspace, so I would like the equivalent of moving a file pointer through the file or some other sort of solution as I simulate data in Simulink.
>
> My data is in a .sim file and must be interpreted as signed integers (here in M-code):
> >> fid = fopen('data.sim', 'r'); %open raw data file
> >> A = fread(fid, XX, 'schar'); % get XX samples of data
>
> What I have to do right now is save as much as I can to the workspace and then use a From Workspace block, but if there is a way to input data without having to use any workspace memory, that would be greatly beneficial.
>
> Does anybody have any suggestions?

Well, it's not elegant, but it seems to work:

Use the Embedded MATLAB Function block with a free-running counter as the driver - I use the following code for the Embedded block:
---
function y = fcn(u)
 
y = 0;
eml.extrinsic('fopen')
eml.extrinsic('fseek')
eml.extrinsic('fread')
eml.extrinsic('fclose')
fid = fopen('dataFile', 'r'); %open data file
fseek(fid,u,'bof'); % seek to location
y = fread(fid,1,'schar');
fclose('all');
-----

You could probably make it faster by taking more than a single sample at a time and then serializing it later. Please comment if there's a better way.

Subject: Simulink: Importing Very Large Data Sets

From: Phil Goddard

Date: 16 Sep, 2009 05:43:04

Message: 3 of 4

There are at least 3 alternative (and in my opinion better) approaches:

a) If you insist on using an EML block then at least make fid a persistent variable.
This way you don't need to open the file each time step.
u could also be a persistent local variable (rather than an input) that gets incremented each time the block is called -- this way you don't need fseek.

The problem with the above approach (and the use of EML to do this type of thing) is that there is no real way to know when the simulation is finishing and hence when to call fclose.

b) Using an S-function (either m-code or c-code).
This has the advantage of having clearly defined bits of the code for initialization and termination.
The termination part inparticular overcomes the above problem in using the EML block.

c) Set up a MATLAB script that reads in a chunk of your file and uses the sim function to run the simulation with that data.
Set up your model to save its final state, then the next time through the loop read in a new chunk of data and once again call the sim function to run the simulation but start it with the previous final state values.

Phil.

Subject: Simulink: Importing Very Large Data Sets

From: Luke

Date: 16 Sep, 2009 14:16:02

Message: 4 of 4

Excellent, thank you Phil - that's exactly what I was looking for.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us