MATLAB Answers

Processing Tall Arrays Taking Too Long

18 views (last 30 days)
Avinash Rajendra
Avinash Rajendra on 1 Nov 2017
I have a tall array with about 1.7 billion rows of data and 14 columns. I want to be able to process this data in the same way that several examples (with airline data) do it. I am just trying to extract one column and find the mean. My code is something like:
ds = datastore('some-file.csv');
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V; %Mx1 tall double %(M should be the same as above)
m = mean(a); %One integer
gather_m = gather(m);
The gather step is taking way too much time. I haven't seen it complete at all. In the examples I have seen, this step is shown to be completed in a few seconds. Eventually, I want to be able to make calculations and plots, but I want to start by making this simple step work first. Can anyone recognize the problem and recommend a solution? I have parallel pool turned on and there are two workers.
Thank you very much.
  2 Comments
Avinash Rajendra
Avinash Rajendra on 1 Nov 2017
They do, but it still takes way too long to run. I feel like I'd be in good shape if the time to run the program was more manageable.

Sign in to comment.

Answers (1)

Kojiro Saito
Kojiro Saito on 2 Nov 2017
It may speed up by configuring read size of datastore. You can know the default read size by
ds.ReadSize
This is the data size which MATLAB reads from the file at one time. You can set higher size than your default and this will reduce file I/O. Please add ds.ReadSize setting, for example,
ds = datastore('some-file.csv');
ds.ReadSize = 100000; % Or higher
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V; Mx1 tall double %(M should be the same as above)
m = mean(a); %One integer
gather_m = gather(m);
Hope this help.
  4 Comments
Dominique Ingala
Dominique Ingala on 7 Apr 2021
I came from Python and R... Same struggle. So I'm now trying Matlab... If you managed to fix this, please share some secrets. Thanks.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!