Thread Subject: MATLAB Reshape Challenge: N x d matrix to N/K x d x K matrix

Hi folks,

I'm working on parallelizing some machine learning code in MATLAB. I'm
using the Parallel Computing Toolbox and the parfor construct;
therefore, I have certain restrictions on how I must "slice" into
certain data structures accessed within the parfor (parallel for)
loop.

Basically, I need to reshape my training data from an (N x d) two-
dimensional matrix to a fully "slicable" (N/K x d x K) matrix, where K
is the number of available parallel threads (which corresponds to the
number of "slicable" input data partitions). Here, I would slice among
the third dimension, i.e., partitioned_data
(:,:,parallel_loop_index_i).

(For now, assume N is a multiple of K.)

I'm usually very good with reshape, permute, shiftdim, etc., but I am
having trouble making this work nicely.

I could hack in something like slicable_data_partitions = cat(3,
manual_partition_1, manual_partition_2, ..., manual_partition_K) but
that's horrible and I was hoping for an efficient one-liner with some
combination of reshape, permute, etc. Performance is important, as
this will be fairly large scale, i.e., numel(data) on the order of
10^7.

Note I can actually get reshape to return output int he desired
dimension, however, the d data elements along the second dimension in
each of the N rows are no longer in the correct order.

Detailed example of desired output:

[N x d, 8 x 3]:

input_data = reshape(1:24, 3, 8)' % Note transpose

input_data =

     1 2 3
     4 5 6
     7 8 9
    10 11 12
    13 14 15
    16 17 18
    19 20 21
    22 23 24


For K = 4, [N/K x d x K, 2 x 3 x 4]

desired_transformed_data =

ans(:,:,1) =

     1 2 3
     4 5 6

ans(:,:,2) =

    7 8 9
    10 11 12

ans(:,:,3) =

    13 14 15
    16 17 18

ans(:,:,4) =

     19 20 21
    22 23 24


Any help would be greatly appreciated.

On a final note, from a memory performance standpoint: any performance
difference in the dimension along which the data is sliced? I.e.,
performance difference for [N/K x d x K] and slicing along the third
dimension, versus [K x d x N/K] and slicing along the first dimension?


With thanks,

--Mike

mprocopio@gmail.com wrote in message <f6776e56-5088-4685-85ee-fa28328610e6@v4g2000yqa.googlegroups.com>...
> Hi folks,
>
> I'm working on parallelizing some machine learning code in MATLAB. I'm
> using the Parallel Computing Toolbox and the parfor construct;
> therefore, I have certain restrictions on how I must "slice" into
> certain data structures accessed within the parfor (parallel for)
> loop.
>
> Basically, I need to reshape my training data from an (N x d) two-
> dimensional matrix to a fully "slicable" (N/K x d x K) matrix, where K
> is the number of available parallel threads (which corresponds to the
> number of "slicable" input data partitions). Here, I would slice among
> the third dimension, i.e., partitioned_data
> (:,:,parallel_loop_index_i).
>
> (For now, assume N is a multiple of K.)
>
> I'm usually very good with reshape, permute, shiftdim, etc., but I am
> having trouble making this work nicely.
>
> I could hack in something like slicable_data_partitions = cat(3,
> manual_partition_1, manual_partition_2, ..., manual_partition_K) but
> that's horrible and I was hoping for an efficient one-liner with some
> combination of reshape, permute, etc. Performance is important, as
> this will be fairly large scale, i.e., numel(data) on the order of
> 10^7.
>
> Note I can actually get reshape to return output int he desired
> dimension, however, the d data elements along the second dimension in
> each of the N rows are no longer in the correct order.
>
> Detailed example of desired output:
>
> [N x d, 8 x 3]:
>
> input_data = reshape(1:24, 3, 8)' % Note transpose
>
> input_data =
>
> 1 2 3
> 4 5 6
> 7 8 9
> 10 11 12
> 13 14 15
> 16 17 18
> 19 20 21
> 22 23 24
>
>
> For K = 4, [N/K x d x K, 2 x 3 x 4]
>
> desired_transformed_data =
>
> ans(:,:,1) =
>
> 1 2 3
> 4 5 6
>
> ans(:,:,2) =
>
> 7 8 9
> 10 11 12
>
> ans(:,:,3) =
>
> 13 14 15
> 16 17 18
>
> ans(:,:,4) =
>
> 19 20 21
> 22 23 24
>
>
> Any help would be greatly appreciated.
>
> On a final note, from a memory performance standpoint: any performance
> difference in the dimension along which the data is sliced? I.e.,
> performance difference for [N/K x d x K] and slicing along the third
> dimension, versus [K x d x N/K] and slicing along the first dimension?
>
>
> With thanks,
>
> --Mike

Hi Mike

Try the following:

>> input_data = [1 2 3
       4 5 6
       7 8 9
       10 11 12
       13 14 15
       16 17 18
       19 20 21
       22 23 24];
>> [N d] = size(input_data);
>> K = 4;
>> permuted_data = permute(reshape(permute(input_data,[2 1]),[d N/K K]),[2 1 3])

HTH,
Nick

Awesome, Nick, thank you so much.

That completely solved the problem, and appears very efficient (0.25s
processing time for K=32, d=87, and N=105504).


Thanks again--I very much appreciate it!

--Mike