MATLAB slows down while saving large amounts of .mat files

I wrote a sub-automated script that triggers a Lecroy oscilloscope to capture waveforms an then save the waveforms into .mat files in a pre-allocated folder. Every time the scope triggers, the waveforms are saved. When I run the script for some 100 or so trigger events, no significant decrease in the saving is observed. However, when I increase the total number of trigger events to some 700 or so, it is observed that MATLAB takes a long time to save the .mat file and then move on to the next trigger. Is this some issue related to how MATLAB allocates RAM? Are there any suggestions on how to improve the speed?

[DGM]

Are the old arrays being cleared from the workspace (or overwritten) once they're written out? How much data are we talking about?

[Howe Li Yeo]

Each .mat file is 110kB and contains waveform data from the scope. For 700 files, the total size becomes 75MB. I've attached a screenshot of the folder for a better understanding. When my script runs, the first 100 or so .mat files can be extracted and saved quite quickly. It takes about 1s between triggering the scope and for the data to be saved on my computer. But at the 500th or so test, the time between iterations slows down quite a lot. It increases to as high as over 20s.

Does clearing the workspace after saving in between iterations help?

Answer 1

[Walter Roberson]

If you might be saving more than 2 gigabytes of data in a single .mat, then you must use -v7.3 format -- and that format can be much slower.

The slowness can occur even for smaller files if you explicitly use the -v7.3 option, or if your preferences are set to use -v7.3 automatically.

Generally speaking, large purely numeric arrays are significantly less affected than composite variables such a struct arrays or cell or objects. If I understand the internals correctly, then a struct scalar with one level of fields that are all numeric, should be reasonably fast.

Another potential challenge is that by default both -v7 and -v7.3 want to compress dat; if your data is hardly compressable then that can waste time.

If you are adding new variables to a -v7 mat file each time, the performance should not be bad. If you use a -v7 file in -append mode and overwrite existing variables, performance can get bad.

If you are using a -v7.3 file and using matfile() to update parts of variables, performance can slow down.

[Howe Li Yeo]

Thank you very much Walter for your detailed answer. My data is not big. My waveforms are saved in .mat files about 110kB each. So 700 of them take up about 75MB total. When my script runs, the first 100 or so .mat files can be extracted and saved quite quickly. It takes about 1s between triggering the scope and for the data to be saved on my computer. But at the 500th or so test, the time between iterations slows down quite a lot. It increases to as high as over 20s.