safeMap

Run long tasks without worrying about unexpected interruptions or memory issues (arrayfun/cellfun alternative)

safeMap saves on disk data generated by long and heavy computations so that you don't have to worry about memory problems and unexpected interruptions.

The passed function is fed with each input cell and the outputs are saved in a file as they are generated; safeMap can either return the file handle or the whole data (with config.returnData, which also deletes the whole file after returning the data). If something interrupts the execution, safeMap will automatically resume once executed again.

Each output of the function can be saved in a cell matrix with the same structure of the input one, or joined (with config.joinUniformOutput). In the latter case, which is the default, each output matrix will be equal to the function outputs stacked along the leading dimensions (e.g. if inputs is of size [3, 3] and each output is of size [4, 2] the final output will be of size [3, 3, 4, 2]).

Moreover, the used file paths and the name under which each output variable is stored can be specified in the configuration.

** Example **

>> output = safeMap(@(x) rand(1,300)*x^5*rand(300,1), repmat({rand(300)}, 100, 100));
Progress: 731 / 10000 (7.3%)
Estimated remaining time: 143s
🔴(Ctrl+C) Operation terminated by user during ...
>> output = safeMap(@(x) rand(1,300)*x^5*rand(300,1), repmat({rand(300)}, 100, 100));
Resuming previous work, from 732-th input
Progress: 10000 / 10000 (100.0%)
Estimated remaining time: 0s

** Limitations **
safeMap writes data on the disk after every output computed, so it's inherently slower than cellfun. Moreover, currently it accepts only one input, though this can be easily overcome like in the example below.

** Check the github readme at the link for more examples **