Applying similar functions to many small chunks of large matrices

I have a large matrix and want to apply similar functions across many smaller portions of the matrix. The matrix I am working with has billions of datapoints, so I am building a smaller replica to practice with.

An example of the practice matrix would be...

T = 5;

W = zeros(10,5);

for trial = 1:T

mtest = randi([1,5],10,3);

W(:,trial) = var(mtest,0,2);

end

M = W(:)

lindex = 0:0.2:1.8;

linc = zeros(50,10);

for i = 1:length(lindex)

linc(:,i) = M >= lindex(i);

end

So, you can see this is a 50X10 matrix of 0's and 1's. Each subset that I want to analyze is a 2X1 chunk of the 50X10 matrix (e.g., linc(1:2,1), linc(1:2,2), linc(1:2,3)...all the way to linc(49:50,10)). If I want to do something simple like calculate the percentage of 1's in each subset, that would require 250 calculations. Is there a function I can write to perform such a calculation in one or a few steps?

I have considered for loops, while loops, and splitapply, but none of them seem appropriate here, though I may be wrong.

Any suggestions?

Thank you in advance!

Answer 1

[Ameer Hamza]

See blockproc: https://www.mathworks.com/help/images/ref/blockproc.html. You can apply same function on smaller blocks of a matrix. For example

M = rand(6) > 0.5; % random binary matrix
blockproc(M, [1 2], @(x) sum(x.data==1))

This statement will count the number of ones in every 1x2 block of matrix M.

[May_the_degrees_of_freedom_be_with_you]

Wow! I never would have thought to look at functions related to image processing. Genius. Thank you so much for your swift help.

I am a Matlab novice, so some of this escapes me. Specifically, I'm not sure what "@(x)" is doing. Based on the below forum, it looks like this is being used as a handle to pass into later functions. So, I can reference (x) whether I want to sum as you did here, or later multiply, divide, etc. Is this correct?

https://www.mathworks.com/matlabcentral/answers/302938-what-does-this-command-x-function-means-in-matlab

[Ameer Hamza]

The @(x) notation is used to create an anonymous function. If you are familiar with other programming languages, you can think of its as a lambda function. It allows for creating a simple function without creating a new .m file.

The third input to blockproc is a function handle. Note that it can either be an anonymous function or a regular function. For example, you can create a .m file like below

function y = myFunction(x)
    % do some processing on x and return the value in y
end

and call the blockproc like this

blockproc(M, [1 2], @myFunction)

This allows creating more complex functions that cannot be done with an anonymous function.

Now you need to consider some requirements before creating the function handle for blockproc. blockproc divides the input matrix M into blocks of size 1x2 and creates struct using those blocks. You can see the field of that struct here: https://www.mathworks.com/help/images/ref/blockproc.html#d118e16538. blockproc then inputs that struct to the function handle (myFunction) https://www.mathworks.com/help/images/ref/blockproc.html#d118e16028. Therefore, in my code above, I used x.data, because data is a field in the struct which blockproc pass to the function handle.

[May_the_degrees_of_freedom_be_with_you]

This makes a lot more sense now. Thank you for your detailed help!