Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Help averaging multidimensional array

Subject: Help averaging multidimensional array

From: David

Date: 20 Feb, 2012 15:09:12

Message: 1 of 15

Hello,

I have a multidimensional array:

gulf 60x90x456

60 and 90 refer to longitude and latitude, 456 is months. each long/lat point has a different value at each month: eg.

gulf(24,53,1) = -84.7

I need to average these values to 10 degree boxes; so the new dataset will become:

new 6x9x456

Obviously using the basic mean function doesnt work as this only produces means of each column therefor mean2 would be needed however the dataset includes many NaN values and mean2 does not account for this and as far as I am aware there is no nanmean equivalent for mean2.
I assume a for-loop would be the most sensible way of doing this but everything I have tried doesn't seem to work.

Any help would be hugely appreciated as this is very important!

Thanks

Subject: Help averaging multidimensional array

From: ImageAnalyst

Date: 20 Feb, 2012 15:15:22

Message: 2 of 15

David:
It's a little tricky and complicated but you could try blockproc() -
that could be the most efficient.

 Another option would be to just get a mean of everything and
subsample every 10th voxel. You could use convn() for that. It
slides a cube along getting the mean inside the cube at every single
location. I'm not sure how convn() handles NaNs - the help doesn't
mention NaNs so you'd just have to experiment and find out.
ImageAnalyst

Subject: Help averaging multidimensional array

From: David

Date: 20 Feb, 2012 15:39:10

Message: 3 of 15

ImageAnalyst <imageanalyst@mailinator.com> wrote in message <b3f2fe20-b746-4514-8ce7-0e9fb877599b@x19g2000yqh.googlegroups.com>...
> David:
> It's a little tricky and complicated but you could try blockproc() -
> that could be the most efficient.
>
> Another option would be to just get a mean of everything and
> subsample every 10th voxel. You could use convn() for that. It
> slides a cube along getting the mean inside the cube at every single
> location. I'm not sure how convn() handles NaNs - the help doesn't
> mention NaNs so you'd just have to experiment and find out.
> ImageAnalyst

Hi,

the blockproc() option is a good idea and i have managed to get it to work for a scaled down array:

test=10x10x1

The code i used is as follows:

fun = @ (block_struct) mean2(block_struct.data);
    new = blockproc(test,[10 10],fun);

This does not give me any errors however unfortunately it still doesnt help the NaN situation. I am wondering if there is any way of using the function;

mean2(x(isfinite(x)));

in this code? I have tried to fit it in but i get errors back.

Cheers

Subject: Help averaging multidimensional array

From: Roger Stafford

Date: 20 Feb, 2012 20:43:11

Message: 4 of 15

"David " <sum_is_41@hotmail.com> wrote in message <jhtnmo$ge3$1@newscl01ah.mathworks.com>...
> I have a multidimensional array:
> gulf 60x90x456
> 60 and 90 refer to longitude and latitude, 456 is months. each long/lat point has a different value at each month: eg.
> gulf(24,53,1) = -84.7
> I need to average these values to 10 degree boxes; so the new dataset will become:
> new 6x9x456
> ...... however the dataset includes many NaN values .......
- - - - - - - -
  I think you can make the 'nanmean' routine work in your task if you organize things in the following manner:

 a. Reshape to a 10 by 6 by 10 by 9 by 456 size.
 b. Use 'permute' to obtain a 10 by 10 by 6 by 9 by 456 size.
 c. Reshape this to a 100 by 6 by 9 by 456 size.
 d. Use 'nanmean' along the first dimension, getting a 1 by 6 by 9 by 456 array of means.
 e. Do a 'squeeze' down to a 6 by 9 by 456 size.

You can actually accomplish all this in one glorious line of code.

  (Note: I haven't tried this - you had better run a validity check on it before trusting it.)

Roger Stafford

Subject: Help averaging multidimensional array

From: David

Date: 21 Feb, 2012 17:19:11

Message: 5 of 15

Thanks a lot guys, I'm sure one of these methods will work eventually.
I appreciate the help.

Dave

Subject: Help averaging multidimensional array

From: Matt J

Date: 21 Feb, 2012 20:27:10

Message: 6 of 15

"David " <sum_is_41@hotmail.com> wrote in message <jhtnmo$ge3$1@newscl01ah.mathworks.com>...
>
> Obviously using the basic mean function doesnt work as this only produces means of each column therefor mean2 would be needed however the dataset includes many NaN values and mean2 does not account for this and as far as I am aware there is no nanmean equivalent for mean2.
> I assume a for-loop would be the most sensible way of doing this but everything I have tried doesn't seem to work.
================

If you don't have blockproc (or even if you do), this might be a good alternative

http://www.mathworks.com/matlabcentral/fileexchange/35085-split-nd-array-into-equal-sized-cells

In your case,

tic;
gulfCell=mat2tiles(gulf,[10,10,1]);
new2=cellfun(@(c) nanmean(c(:)),gulfCell);
toc

Subject: Help averaging multidimensional array

From: Matt J

Date: 21 Feb, 2012 22:30:12

Message: 7 of 15

"Matt J" wrote in message <ji0umu$g0$1@newscl01ah.mathworks.com>...
> "David " <sum_is_41@hotmail.com> wrote in message <jhtnmo$ge3$1@newscl01ah.mathworks.com>...
> >
> > Obviously using the basic mean function doesnt work as this only produces means of each column therefor mean2 would be needed however the dataset includes many NaN values and mean2 does not account for this and as far as I am aware there is no nanmean equivalent for mean2.
> > I assume a for-loop would be the most sensible way of doing this but everything I have tried doesn't seem to work.
> ================
>
> If you don't have blockproc (or even if you do), this might be a good alternative
>
> http://www.mathworks.com/matlabcentral/fileexchange/35085-split-nd-array-into-equal-sized-cells
>
> In your case,
>
> tic;
> gulfCell=mat2tiles(gulf,[10,10,1]);
> new2=cellfun(@(c) nanmean(c(:)),gulfCell);
> toc



Here's another, much faster way:


gulf=rand(60,90,456);
nanmap=isnan(gulf);
gulf(nanmap)=0;

tic;
 t=reshape(gulf,10,6,10,9,[]);
 t=sum(sum(t,1),3);
 
 nm=reshape(nanmap,10,6,10,9,[]);
 nm=sum(sum(nm,1),3);
 
 new=reshape(t./nm,6,9,456);
 
toc

Subject: Help averaging multidimensional array

From: David

Date: 22 Feb, 2012 16:33:10

Message: 8 of 15

"Matt J" wrote in message <ji15tk$pce$1@newscl01ah.mathworks.com>...
> "Matt J" wrote in message <ji0umu$g0$1@newscl01ah.mathworks.com>...
> > "David " <sum_is_41@hotmail.com> wrote in message <jhtnmo$ge3$1@newscl01ah.mathworks.com>...
> > >
> > > Obviously using the basic mean function doesnt work as this only produces means of each column therefor mean2 would be needed however the dataset includes many NaN values and mean2 does not account for this and as far as I am aware there is no nanmean equivalent for mean2.
> > > I assume a for-loop would be the most sensible way of doing this but everything I have tried doesn't seem to work.
> > ================
> >
> > If you don't have blockproc (or even if you do), this might be a good alternative
> >
> > http://www.mathworks.com/matlabcentral/fileexchange/35085-split-nd-array-into-equal-sized-cells
> >
> > In your case,
> >
> > tic;
> > gulfCell=mat2tiles(gulf,[10,10,1]);
> > new2=cellfun(@(c) nanmean(c(:)),gulfCell);
> > toc
>
>
>
> Here's another, much faster way:
>
>
> gulf=rand(60,90,456);
> nanmap=isnan(gulf);
> gulf(nanmap)=0;
>
> tic;
> t=reshape(gulf,10,6,10,9,[]);
> t=sum(sum(t,1),3);
>
> nm=reshape(nanmap,10,6,10,9,[]);
> nm=sum(sum(nm,1),3);
>
> new=reshape(t./nm,6,9,456);
>
> toc


Unfortunately i have the matlab version 7.10.0 which doesn't have the function mat2tiles.
I tried the second code you gave me, it doesn't return any errors and the array does come out 6x9x456 correctly, however when i view the data it is all wrong so not quite sure if thats the right way to go.
Just to make sure, the code i would use would be as follows:

nanmap=isnan(gulf);
gulf(nanmap)=0;
tic;
t=reshape(gulf,10,6,10,9,[]);
t=sum(sum(t,1),3);
nm=reshape(nanmap,10,6,10,9,[]);
nm=sum(sum(nm,1),3);
new=reshape(t./nm,6,9,456);
toc;

Just making sure I havn't made a mistake.

Subject: Help averaging multidimensional array

From: David

Date: 22 Feb, 2012 16:56:12

Message: 9 of 15

> I think you can make the 'nanmean' routine work in your task if you organize things in the following manner:
>
> a. Reshape to a 10 by 6 by 10 by 9 by 456 size.
> b. Use 'permute' to obtain a 10 by 10 by 6 by 9 by 456 size.
> c. Reshape this to a 100 by 6 by 9 by 456 size.
> d. Use 'nanmean' along the first dimension, getting a 1 by 6 by 9 by 456 array of means.
> e. Do a 'squeeze' down to a 6 by 9 by 456 size.
>
> You can actually accomplish all this in one glorious line of code.
>
> (Note: I haven't tried this - you had better run a validity check on it before trusting it.)
>
> Roger Stafford

Thanks Roger, i tried your method and it has worked in a way, all of the values have been averaged correctly, but the only problem is that i need all of my NaN values to maintain their shape (which i should have mentioned before).

Because they represent land; so if you can picture what the contour map would look like at one month, it is land round the side with all of the values in the middle (N.Atlantic).
 So i somehow need a way of doing exactly what you have given me but without affecting the NaNs...
As now the land looks like jagged triangles.

Any thoughts would be appreciated.

Thanks a lot.
Dave

Subject: Help averaging multidimensional array

From: Matt J

Date: 22 Feb, 2012 16:58:15

Message: 10 of 15

"David " <sum_is_41@hotmail.com> wrote in message <ji35c6$9e2$1@newscl01ah.mathworks.com>...
>
> Unfortunately i have the matlab version 7.10.0 which doesn't have the function mat2tiles.
==================

You were meant to download MAT2TILES from the link I gave you earlier :)



> nm=reshape(nanmap,10,6,10,9,[]);
================

Sorry, small error in this line. It should be

nm=reshape(~nanmap,10,6,10,9,[]); %note the ~nanmap

Subject: Help averaging multidimensional array

From: David

Date: 22 Feb, 2012 16:59:15

Message: 11 of 15

I have just realised the reason it looks wrong is not because the code is wrong. What i failed to mention before is that i need the NaN values to maintain their shape as they represent land on a contour map.

Using this code a lot of the NaN values are removed making the contour map not make sense at all.

Can you think of any way of using this code for all of the values without affecting the shape of the NaNs?

Thanks for your help
Dave

Subject: Help averaging multidimensional array

From: David

Date: 22 Feb, 2012 17:09:12

Message: 12 of 15

"Matt J" wrote in message <ji36r7$et4$1@newscl01ah.mathworks.com>...
> "David " <sum_is_41@hotmail.com> wrote in message <ji35c6$9e2$1@newscl01ah.mathworks.com>...
> >
> > Unfortunately i have the matlab version 7.10.0 which doesn't have the function mat2tiles.
> ==================
>
> You were meant to download MAT2TILES from the link I gave you earlier :)
>
>
>
> > nm=reshape(nanmap,10,6,10,9,[]);
> ================
>
> Sorry, small error in this line. It should be
>
> nm=reshape(~nanmap,10,6,10,9,[]); %note the ~nanmap

Ohh sorry i completely missed that!
Ive downloaded the code...the question now is how do i use it?
Sorry to take up your time
Cheers.

Subject: Help averaging multidimensional array

From: David

Date: 22 Feb, 2012 17:14:13

Message: 13 of 15

Ignore the last message I was just being an idiot.

I've tried the mat2tiles method but it has the same problem as the other method.
So the code does do what it is supposed to but the NaN values are messed up so the contour plot loses all shape.

Not sure how im gonna get round this one!

Subject: Help averaging multidimensional array

From: Matt J

Date: 22 Feb, 2012 19:40:13

Message: 14 of 15

"David " <sum_is_41@hotmail.com> wrote in message <ji37p5$htp$1@newscl01ah.mathworks.com>...
> Ignore the last message I was just being an idiot.
>
> I've tried the mat2tiles method but it has the same problem as the other method.
> So the code does do what it is supposed to but the NaN values are messed up so the contour plot loses all shape.
>
> Not sure how im gonna get round this one!

If you have too much missing data, then there may not be much you can do. Maybe you should be filling in the NaNs with interpolated data, e.g., using

http://www.mathworks.com/matlabcentral/fileexchange/27994-inpaint-over-missing-data-in-n-d-arrays

before you do the downsampling.

Subject: Help averaging multidimensional array

From: Roger Stafford

Date: 22 Feb, 2012 20:27:12

Message: 15 of 15

"David " <sum_is_41@hotmail.com> wrote in message <ji36nc$ejo$1@newscl01ah.mathworks.com>...
> Thanks Roger, i tried your method and it has worked in a way, all of the values have been averaged correctly, but the only problem is that i need all of my NaN values to maintain their shape (which i should have mentioned before).
>
> Because they represent land; so if you can picture what the contour map would look like at one month, it is land round the side with all of the values in the middle (N.Atlantic).
> So i somehow need a way of doing exactly what you have given me but without affecting the NaNs...
> As now the land looks like jagged triangles.
- - - - - - -
  You should be aware that the NaNs which survived after your processing are not any of those that were present earlier but only indicate 10 by 10 blocks in which there were no finite numbers at all and are nanmean's way of expressing displeasure at not being able to produce a meaningful mean value. Your "jagged triangles" are therefore remnants of 10 by 10 blocks of solid NaNs, one for each such block.

  If you want your "land" to survive in a more evident form, I think you need to modify your requirements so as to, say, allow a block with a sufficient number of NaNs, (say 50,) to produce a NaN result. Or perhaps a more sophisticated algorithm which senses that a sufficiently large contiguous collection of NaNs are present in the block.

Roger Stafford

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us