Outer product of tall arrays

I have two column vectors x and y which I cast as tall arrays. I want to take their outer product x*y', however I cannot transpose a tall array. What other way is there to perform the computation x*y'?

[Torsten]

however I cannot transpose a tall array.

Why ? You can transpose every array.

By the way:

Usual transpose is

.' 

, not

'

[Z]

A tall array cannot be transposed. It cannot even be reshaped to a row vector.

[John D'Errico]

This makes little sense, as you cannot then get a result in any way.

If X and Y are column vectors, then X*Y' wil be a huge array, too larege to represent in memory. If they are really so large that X and Y need themselves to be stored as tall, then the outer product will be immense. You don't have sufficient disk space to store the result. If you were taking an INNER product, the result would be a scalar, so there would be no problem. (The INNER product would be trivial to compute for tall arrays. But you explicitly asked for an outer product.)

[Torsten]

I didn't know that "Tall array" really is a terminus technicus:

https://de.mathworks.com/help/matlab/tall-arrays.html

If you only need certain elements of z = x*y.', you can do it of course elementwise:

z(i,j) = x(i)*y(j)

[Z]

@John D'Errico See for yourself.

>> x=tall([1;2;3])

Starting parallel pool (parpool) using the 'local' profile ...

Connected to the parallel pool (number of workers: 8).

x =

3×1 tall double column vector

1

2

3

>> x'

Error using '

ctranspose does not support tall arrays.

>> x.'

Error using .'

transpose does not support tall arrays.

I can do the dot product using dot() so I avoid transposing.

[Torsten]

https://de.mathworks.com/matlabcentral/answers/406289-how-to-take-transpose-of-a-tall-array

[Z]

@John D'Errico Actually in my case, the column vectors can be stored in memory, I just need to cast them to tall arrays because the outer product cannot. So the fix would be to transpose the second vector before casting.

UPDATE: This didn't work because MATLAB does not allow multiplication of two tall arrays unless one is a scalar. I guess this has to do with what you pointed out.

[John D'Errico]

@Z

NO. You did not listen to me. Think about what you are asking to do. If your arrays really are so tiny, then just convert them to regular, boring vectors. Then compute the outer product. Of course then, why in the name of god and little green apples are you using tall arrays in the first place if they are small?

But just suppose that your vectors REALLY do need to be tall arrays, as they are too large to store in memory. That is why you would use a tall array. For example, perhaps x and y are each 1e9 elements long.

Then the trivially simple rank 1 array the outer product would generate has 1e9*1e9=1e18 elements, each of which uses 8 bytes of RAM to store. Do you have that much disk storage available, so around 8e18 bytes, or roughly 8e9 GIGABYTES? That would be 8e6 terabytes. Or 8e3 petabytes. Or roughly 8 exabytes.

Do you, honestly, seriously have an 8 exabyte drive? Probably more than that of course, as things are never quite that easy. I suppose, if you work for the NSA, and you have full access to some seriously massive computers, you might have it. But then, if you do work there, you surely do not need me to explain this!

The computation you want to perform would have an immense result. You CANNOT DO IT, EVEN IF YOU COULD TAKE THE TRANSPOSE.

[John D'Errico]

Suppose the column vectors are small enough, that you can trivially store them in memory.

So say that x and y are only 1e7 elements in length. Then x*y' would use only 8e14 bytes of memory to store.

This is still a huge array, 800 terabytes.

[Z]

@John D'Errico I see. Thank you for your responses!