.mat files taking a very long time to load

I have a script which needs to work through many large .mat files (all about 17 MB) - one for each day of the years from 2000 to 2019 - and then to do so 120 times for another set of conditions. This takes a very long time if I simply use load() to do so. I have tried converting to .zip files, converting to another version of .mat file (uncompressed with v6 or compressed with v7.3 and using the matfile() function) but to no avail. I have also tried downloading them all once (rather than the 120 times) and saving to a cell array for later access, but the array soon grew too large and matlab crashed.
Is there a faster way to do something like this? Any help would be much appreciated.

4 Comments

You probably can't avoid the slow disk (SSDs are also slow when you compare them to RAM).
The only high-level advice I can give you is to limit the number of times you load the same file as much as you can.
Rik, would working with an hdf5 file be faster at all? The mat files are derived from original hdf5 files but I still have the latter on hand.
Rik
Rik on 12 Jun 2019
Edited: Rik on 12 Jun 2019
The mat files might have a slight edge (because they probably require less processing to load). I doubt you will succeed in getting this to work fast. You could possibly get it to run faster, maybe with a RAID setup?
Thanks for your help. Looks like I might just have to stick with the way it is currently running and just bear it, but I'll look around.

Sign in to comment.

Answers (0)

Categories

Find more on MATLAB in Help Center and File Exchange

Asked:

on 12 Jun 2019

Commented:

on 12 Jun 2019

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!