How can I run the kernels in parallel using "parallel.​gpu.CUDAKe​rnel" from Matlab

I have a small toy code below. I intend to launch the 4 kernels in parallel. The idea behind is that, I will be operating on large 3D matrices. I intend to write a kernel that will operate on a small chunk of that 3D matrix. Is there a way I can launch multiple of these kernels in parallel instead of serially one after the other in this example.

//////////////////////////////////

#include < cuda.h >

_global_ void add_gpu_kernel(double *c, const double *a, const double *b) {

    int x =  threadIdx.x;
    c[x] = a[x] + b[x];
}
//////////////////////////////////

%//////////////////////////////////

gd = gpuDevice();

A1 = gpuArray(double(randn(N,N,'double')));

B1 = gpuArray(double(randn(N,N,'double')));

C1 = gpuArray(double(zeros(N,N,'double')));

C2 = gpuArray(C1);

N=5*2; A1 = gpuArray(double(randn(N,N,'double')));

B1 = gpuArray(double(randn(N,N,'double')));

C1 = A1+B1;

kernel = parallel.gpu.CUDAKernel('add_gpu_kernel.ptx', 'add_gpu_kernel.cu');

kernel.SharedMemorySize=1;

kernel.ThreadBlockSize = [5*5,5*5,1];

kernel.GridSize = [2,2,1];

tic;

C2(1:5,1:5)=kernel.feval(C2(1:5,1:5),A1(1:5,1:5),B1(1:5,1:5));

C2(6:10,1:5)=kernel.feval(C2(6:10,1:5),A1(6:10,1:5),B1(6:10,1:5));

C2(1:5,6:10)=kernel.feval(C2(1:5,6:10),A1(1:5,6:10),B1(1:5,6:10));

C2(6:10,6:10)=kernel.feval(C2(6:10,6:10),A1(6:10,6:10),B1(6:10,6:10));

wait(gd);

time=toc;

error=sum(sum(sum(abs(C1-C3))));

%//////////////////////////////////

[Joss Knight]

CUDAKernels are not synchronous, which means they can run in parallel in theory. But you are writing to C2 on one line and reading from it again the next, which means that MATLAB will wait for the first kernel to finish before launching the next one. The solution is to give each chunk of the matrix a different variable name and assign the results into the output at the end (you could use cell arrays for instance).