Clear Filters
Clear Filters

Is there a parallel version of splitapply()?

5 views (last 30 days)
Group-based computation is naturally sutiable for parallel computation. I wonder why Matlab has not yet a built-in parfor-splitapply? In Github, there is a repository called Matlab-FunUtils, which has cmap.m. It's like a parfor version of arrayfun. I couldn't find a similar function for splitapply. I have been trying to modify Matlab's splitapply, but the codes are very difficult to understand. Any suggesstoins? Should I write from scratch?

Accepted Answer

Edric Ellis
Edric Ellis on 15 Sep 2023
tall arrays support parallel execution of splitapply. Would that work for your case?
  6 Comments
Simon
Simon on 20 Sep 2023
@Edric Ellis When my codes have nested splitapply, the tall approach fails. The situation occurs when I have 2-layer nested groupping structure, and splitapply() is used to process groups in each layer. For example, like in patients.mat, suppose there is a new grouping variable, 'BloodType', within each Male/Female group in Gender variable. tall will not allow splitapply to work in both level of groups. I guess this is the intrinsic limitation of parallel toolbox? Similarly, it does not allow nested parfor-loop.
Edric Ellis
Edric Ellis on 20 Sep 2023
Yes, typically this sort of algorithm cannot work in a nested way. It isn't quite the same restriction as nested parfor (in fact, with thread-pools, you can get genuine 2-level parallelism with parfor, but there's a separate restriction that you need to "hide" the inner parfor inside a function).
As to your original question - does it work to make the groups based on the combinations of variables, as per the 2nd syntax of findgroups ? I.e. something like:
load patients
G = findgroups(Smoker, Age >= 40)
G = 100×1
3 2 1 2 2 2 3 2 1 1
This divides the data into 4 groups for each combination of Smoker and Age >= 40.

Sign in to comment.

More Answers (1)

Bruno Luong
Bruno Luong on 15 Sep 2023
To me there are 2 reasons:
  • Not everyone have parallel toolbox
  • The user function can be multi-threaded and already use efficiently the CPU cores, add parallel on top will then likely reduce the performance. The parallel computation should never be applied automatically anywhere. It should be judged case by case.
  2 Comments
Simon
Simon on 15 Sep 2023
I do computations with tall, tall tables. parfor saves huge amount of runtime for me compared with for-looping through rows serially. Now I am doing group-based computations and the number of groups I have is large. For that, I feel that splitapply is a little sluggish.
Simon
Simon on 16 Sep 2023
@Bruno Luong "Not everyone have parallel toolbox"
I know that. I was very hesitant to use parallel toolbox. I thought it would be extremely difficult to adopt parallelization. But I dived in giving it a try. It turned out Matlab has done a wonderful job making a daunting task beginner-easy. I google how parallelization could be done in Python, and again what I find is choices of possible useful packages, new sets of documentations to digest, ... and etc.
IMHO, if one has chosen Matlab, he should seriously consider adopting parallel toolbox. It's easy to use, and combined with group-processing functions and functional programming, to shift the whole coding game to a different paradigm, saving both coding and run time.

Sign in to comment.

Categories

Find more on Preprocessing Data in Help Center and File Exchange

Products


Release

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!