Thread Subject:

preallocating space for a structure?

Hi all. I'd appreciate any input on this...

I've written a program that analyzes a bunch of data and stores some results in a two field structure array. Using the progressbar function (downloaded from file exchange), it was obvious that the beginning iterations of the loop complete much faster than later iterations, which told me that I needed to preallocate a chunk of memory for the structure. I'm pretty sure that I've now preallocated memory correctly, however while the program now runs faster than it did before, the estimated time remaining as the loop progresses continues to increase.

As I mentioned, it's a two field structure. The size of the final structure will be somewhere in the (1,4000) to (1,6000) range. Each element in the structure contains a vector of variable length, say somewhere between (1,100) and (1,1101). 1101 is the maximum possible length of an individual element.

I tried to preallocate space as follows:

example_num = 4000;

for i = 1:example_num
    master(i).mz = zeros(1101,1);
    master(i).int = zeros(1101,1);
end

With each iteration of the major loop in the program, each element of this structure is replaced by it's real vector, using something similar to:

for j = 1:example_num
    %insert a bunch of other code here
    master(j).mz = random_vector1;
    master(j).int = random_vector2;
end

Does this seem like the space is preallocated correctly? I'm confused as to why the time it takes to perform each iteration continues to grow. Thanks!

"Jeremy " <jeremy.brower@asu.edu> wrote in message <htid1t$89l$1@fred.mathworks.com>...
> Hi all. I'd appreciate any input on this...
>
> I've written a program that analyzes a bunch of data and stores some results in a two field structure array. Using the progressbar function (downloaded from file exchange), it was obvious that the beginning iterations of the loop complete much faster than later iterations, which told me that I needed to preallocate a chunk of memory for the structure. I'm pretty sure that I've now preallocated memory correctly, however while the program now runs faster than it did before, the estimated time remaining as the loop progresses continues to increase.
>
> As I mentioned, it's a two field structure. The size of the final structure will be somewhere in the (1,4000) to (1,6000) range. Each element in the structure contains a vector of variable length, say somewhere between (1,100) and (1,1101). 1101 is the maximum possible length of an individual element.
>
> I tried to preallocate space as follows:
>
> example_num = 4000;
>
> for i = 1:example_num
> master(i).mz = zeros(1101,1);
> master(i).int = zeros(1101,1);
> end
>
> With each iteration of the major loop in the program, each element of this structure is replaced by it's real vector, using something similar to:
>
> for j = 1:example_num
> %insert a bunch of other code here
> master(j).mz = random_vector1;
> master(j).int = random_vector2;
> end
>
> Does this seem like the space is preallocated correctly?

No the way you preallocate is completely off.

1) You need to preallocate the STRUCTURE array, not its field.
2) There is no gain to preallocate the fields with ZEROS then erase them later on with RANDOM_VECTOR*.
3) However you could preallocate the field only if you know ahead of time its size. In that case use the command: master(j).mz(:) = random_vector1;

That will not totally prevent your program to slowdown, since the RAM is occupied more when the loop go on.

Bruno

-------
No the way you preallocate is completely off.

1) You need to preallocate the STRUCTURE array, not its field.
2) There is no gain to preallocate the fields with ZEROS then erase them later on with RANDOM_VECTOR*.
3) However you could preallocate the field only if you know ahead of time its size. In that case use the command: master(j).mz(:) = random_vector1;

That will not totally prevent your program to slowdown, since the RAM is occupied more when the loop go on.

Bruno
--------
Bruno - Thanks for the reply. My understanding was that you need to preallocate an array with enough space to account for the final array. If this is true, then preallocating the structure with a bunch of blank arrays does not achieve the purpose. To address your point 3, I did preallocate the field with the maximum possible length. When each element replaces the max possible length of zeros, no harm no foul right? I don't think that your explanation describes why the first 100 iterations of my loop complete in about 1/100th the time of the final 100 iterations. Can you explain in more detail?