<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
  <channel>
    <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947</link>
    <title>MATLAB Central Newsreader - decrease in speed due to appending a row vector</title>
    <description>Feed for thread: decrease in speed due to appending a row vector</description>
    <language>en-us</language>
    <copyright>&amp;copy;1994-2012 by MathWorks, Inc.</copyright>
    <webmaster>webmaster@mathworks.com</webmaster>
    <generator>MATLAB Central Newsreader</generator>
    <docs>http://blogs.law.harvard.edu/tech/rss</docs>
    <ttl>60</ttl>
    <image>
      <title>MathWorks</title>
      <url>http://www.mathworks.com/images/membrane_icon.gif</url>
    </image>
    <item>
      <pubDate>Thu, 24 May 2007 19:44:44 -0400</pubDate>
      <title>decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374801</link>
      <author>Brian</author>
      <description>I have a row vector and I am adding new data elements in between at&lt;br&gt;
different locations. The approach i take is copy the vector in a&lt;br&gt;
temporary variable, add the new element at the desired location and&lt;br&gt;
then append the remaining elements from the temporary variable. I&lt;br&gt;
believed it should be very fast in Matlab but if i repeat it 15000&lt;br&gt;
times it takes me 10 seconds which is very slow for my application.&lt;br&gt;
Is there any other way to get around this problem.</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 20:17:29 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374803</link>
      <author>Joseph</author>
      <description>pre-allocate the space for your vector&lt;br&gt;
&lt;br&gt;
help zeros&lt;br&gt;
&lt;br&gt;
&amp;nbsp;Brian wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; I have a row vector and I am adding new data elements in between at&lt;br&gt;
&amp;gt; different locations. The approach i take is copy the vector in a&lt;br&gt;
&amp;gt; temporary variable, add the new element at the desired location and&lt;br&gt;
&amp;gt; then append the remaining elements from the temporary variable. I&lt;br&gt;
&amp;gt; believed it should be very fast in Matlab but if i repeat it 15000&lt;br&gt;
&amp;gt; times it takes me 10 seconds which is very slow for my application.&lt;br&gt;
&amp;gt; Is there any other way to get around this problem.</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 20:20:24 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374804</link>
      <author>Brian</author>
      <description>I don't know how big will the vector grow each time. The space&lt;br&gt;
depends on the input data which will change most of the time and I&lt;br&gt;
have no way of know the final size.&lt;br&gt;
&lt;br&gt;
Joseph wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; pre-allocate the space for your vector&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; help zeros&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; I have a row vector and I am adding new data elements in&lt;br&gt;
between&lt;br&gt;
&amp;gt; at&lt;br&gt;
&amp;gt;&amp;gt; different locations. The approach i take is copy the vector in&lt;br&gt;
a&lt;br&gt;
&amp;gt;&amp;gt; temporary variable, add the new element at the desired location&lt;br&gt;
&amp;gt; and&lt;br&gt;
&amp;gt;&amp;gt; then append the remaining elements from the temporary variable.&lt;br&gt;
I&lt;br&gt;
&amp;gt;&amp;gt; believed it should be very fast in Matlab but if i repeat it&lt;br&gt;
&amp;gt; 15000&lt;br&gt;
&amp;gt;&amp;gt; times it takes me 10 seconds which is very slow for my&lt;br&gt;
&amp;gt; application.&lt;br&gt;
&amp;gt;&amp;gt; Is there any other way to get around this problem.</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 20:35:36 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374805</link>
      <author>John D'Errico</author>
      <description>Brian wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; I don't know how big will the vector grow each time. The space&lt;br&gt;
&amp;gt; depends on the input data which will change most of the time and I&lt;br&gt;
&amp;gt; have no way of know the final size.&lt;br&gt;
&lt;br&gt;
Take a look at my growdata or growdata2&lt;br&gt;
codes. They came from discussions here&lt;br&gt;
on c.s-s.m, and were written to allow&lt;br&gt;
arrays to grow in size with a minimum&lt;br&gt;
penalty in time. My goal was to allow&lt;br&gt;
arrays to grow in size to potentially&lt;br&gt;
millions of elements, without advance&lt;br&gt;
knowledge of the final count. These codes&lt;br&gt;
use cell arrays, each cell of which is&lt;br&gt;
preallocated to use moderately large&lt;br&gt;
but still manageable chunks of memory.&lt;br&gt;
Arrays with huge numbers of cells can&lt;br&gt;
have problems too.&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=8334&amp;objectType=file&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=8334&amp;objectType=file&lt;/a&gt;&amp;gt;&lt;br&gt;
&lt;br&gt;
Of course, its always better to just&lt;br&gt;
preallocate your large arrays whenever&lt;br&gt;
you can.&lt;br&gt;
&lt;br&gt;
HTh,&lt;br&gt;
John</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 18:18:56 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374806</link>
      <author>NZTideMan</author>
      <description>On May 25, 12:35 pm, &quot;John D'Errico&quot; &amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
wrote:&lt;br&gt;
&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; I don't know how big will the vector grow each time. The space&lt;br&gt;
&amp;gt; &amp;gt; depends on the input data which will change most of the time and I&lt;br&gt;
&amp;gt; &amp;gt; have no way of know the final size.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Take a look at my growdata or growdata2&lt;br&gt;
&amp;gt; codes. They came from discussions here&lt;br&gt;
&amp;gt; on c.s-s.m, and were written to allow&lt;br&gt;
&amp;gt; arrays to grow in size with a minimum&lt;br&gt;
&amp;gt; penalty in time. My goal was to allow&lt;br&gt;
&amp;gt; arrays to grow in size to potentially&lt;br&gt;
&amp;gt; millions of elements, without advance&lt;br&gt;
&amp;gt; knowledge of the final count. These codes&lt;br&gt;
&amp;gt; use cell arrays, each cell of which is&lt;br&gt;
&amp;gt; preallocated to use moderately large&lt;br&gt;
&amp;gt; but still manageable chunks of memory.&lt;br&gt;
&amp;gt; Arrays with huge numbers of cells can&lt;br&gt;
&amp;gt; have problems too.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;  &amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&lt;/a&gt;&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Of course, its always better to just&lt;br&gt;
&amp;gt; preallocate your large arrays whenever&lt;br&gt;
&amp;gt; you can.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; HTh,&lt;br&gt;
&amp;gt; John&lt;br&gt;
&lt;br&gt;
I didn't know about your growdata routines, John, so I have devised&lt;br&gt;
the following way for accumulating results of a Monte Carlo&lt;br&gt;
simulation:&lt;br&gt;
I generate a structure:&lt;br&gt;
s(irun).ymx=a;&lt;br&gt;
where irun is the Monte Carlo run number, and the result of the&lt;br&gt;
simulation a(1,n) is a row vector where n is the number of events&lt;br&gt;
(drawn from a Poisson distribution). n varies from run to run.&lt;br&gt;
I understand that when you do it this way, each array in s uses&lt;br&gt;
contiguous memory, but they are not contiguous within the structure.&lt;br&gt;
Thus, the memory allocation problem does not arise.&lt;br&gt;
I'd appreciate your comments on this.</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 18:37:29 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374808</link>
      <author>NZTideMan</author>
      <description>On May 25, 12:35 pm, &quot;John D'Errico&quot; &amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
wrote:&lt;br&gt;
&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; I don't know how big will the vector grow each time. The space&lt;br&gt;
&amp;gt; &amp;gt; depends on the input data which will change most of the time and I&lt;br&gt;
&amp;gt; &amp;gt; have no way of know the final size.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Take a look at my growdata or growdata2&lt;br&gt;
&amp;gt; codes. They came from discussions here&lt;br&gt;
&amp;gt; on c.s-s.m, and were written to allow&lt;br&gt;
&amp;gt; arrays to grow in size with a minimum&lt;br&gt;
&amp;gt; penalty in time. My goal was to allow&lt;br&gt;
&amp;gt; arrays to grow in size to potentially&lt;br&gt;
&amp;gt; millions of elements, without advance&lt;br&gt;
&amp;gt; knowledge of the final count. These codes&lt;br&gt;
&amp;gt; use cell arrays, each cell of which is&lt;br&gt;
&amp;gt; preallocated to use moderately large&lt;br&gt;
&amp;gt; but still manageable chunks of memory.&lt;br&gt;
&amp;gt; Arrays with huge numbers of cells can&lt;br&gt;
&amp;gt; have problems too.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;  &amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&lt;/a&gt;&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Of course, its always better to just&lt;br&gt;
&amp;gt; preallocate your large arrays whenever&lt;br&gt;
&amp;gt; you can.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; HTh,&lt;br&gt;
&amp;gt; John&lt;br&gt;
&lt;br&gt;
I didn't know about your growdata routines, John, so I have devised&lt;br&gt;
the following way for accumulating results of a Monte Carlo&lt;br&gt;
simulation:&lt;br&gt;
I generate a structure:&lt;br&gt;
s(irun).ymx=a;&lt;br&gt;
where irun is the Monte Carlo run number, and the result of the&lt;br&gt;
simulation a(1,n) is a row vector where n is the number of events&lt;br&gt;
(drawn from a Poisson distribution). n varies from run to run.&lt;br&gt;
I understand that when you do it this way, each array in s uses&lt;br&gt;
contiguous memory, but they are not contiguous within the structure.&lt;br&gt;
Thus, the memory allocation problem does not arise.&lt;br&gt;
I'd appreciate your comments on this.</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 22:50:37 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374816</link>
      <author>John D'Errico</author>
      <description>NZTideMan wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; On May 25, 12:35 pm, &quot;John D'Errico&quot;&lt;br&gt;
&amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
&amp;gt; wrote:&lt;br&gt;
&amp;gt;&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; &amp;gt; I don't know how big will the vector grow each time. The&lt;br&gt;
space&lt;br&gt;
&amp;gt;&amp;gt; &amp;gt; depends on the input data which will change most of the&lt;br&gt;
time&lt;br&gt;
&amp;gt; and I&lt;br&gt;
&amp;gt;&amp;gt; &amp;gt; have no way of know the final size.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Take a look at my growdata or growdata2&lt;br&gt;
&amp;gt;&amp;gt; codes. They came from discussions here&lt;br&gt;
&amp;gt;&amp;gt; on c.s-s.m, and were written to allow&lt;br&gt;
&amp;gt;&amp;gt; arrays to grow in size with a minimum&lt;br&gt;
&amp;gt;&amp;gt; penalty in time. My goal was to allow&lt;br&gt;
&amp;gt;&amp;gt; arrays to grow in size to potentially&lt;br&gt;
&amp;gt;&amp;gt; millions of elements, without advance&lt;br&gt;
&amp;gt;&amp;gt; knowledge of the final count. These codes&lt;br&gt;
&amp;gt;&amp;gt; use cell arrays, each cell of which is&lt;br&gt;
&amp;gt;&amp;gt; preallocated to use moderately large&lt;br&gt;
&amp;gt;&amp;gt; but still manageable chunks of memory.&lt;br&gt;
&amp;gt;&amp;gt; Arrays with huge numbers of cells can&lt;br&gt;
&amp;gt;&amp;gt; have problems too.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; &amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&lt;/a&gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Of course, its always better to just&lt;br&gt;
&amp;gt;&amp;gt; preallocate your large arrays whenever&lt;br&gt;
&amp;gt;&amp;gt; you can.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; HTh,&lt;br&gt;
&amp;gt;&amp;gt; John&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; I didn't know about your growdata routines, John, so I have devised&lt;br&gt;
&amp;gt; the following way for accumulating results of a Monte Carlo&lt;br&gt;
&amp;gt; simulation:&lt;br&gt;
&amp;gt; I generate a structure:&lt;br&gt;
&amp;gt; s(irun).ymx=a;&lt;br&gt;
&amp;gt; where irun is the Monte Carlo run number, and the result of the&lt;br&gt;
&amp;gt; simulation a(1,n) is a row vector where n is the number of events&lt;br&gt;
&amp;gt; (drawn from a Poisson distribution). n varies from run to run.&lt;br&gt;
&amp;gt; I understand that when you do it this way, each array in s uses&lt;br&gt;
&amp;gt; contiguous memory, but they are not contiguous within the&lt;br&gt;
&amp;gt; structure.&lt;br&gt;
&amp;gt; Thus, the memory allocation problem does not arise.&lt;br&gt;
&amp;gt; I'd appreciate your comments on this.&lt;br&gt;
&lt;br&gt;
This is also a reassonable way to solve the&lt;br&gt;
problem.&lt;br&gt;
&lt;br&gt;
Use of a single cell for each new appended&lt;br&gt;
element is another way that is very similar&lt;br&gt;
to the structure approach that you used.&lt;br&gt;
Use of pointers here is more efficient&lt;br&gt;
than growing and re-allocating the entire&lt;br&gt;
array. I'm not sure about use of structures,&lt;br&gt;
as I thought that structures were not&lt;br&gt;
efficient users of memory as compared to&lt;br&gt;
cell arrays. This is probably changing&lt;br&gt;
with each matlab release of course. (See&lt;br&gt;
below for a comparison.)&lt;br&gt;
&lt;br&gt;
An issue is how many total elements that&lt;br&gt;
you may expect to see. In some earlier&lt;br&gt;
testing, I found that cell arrays with&lt;br&gt;
millions of cells caused some serious&lt;br&gt;
problems, however a few thousand, or even&lt;br&gt;
ten thousand or so cells was not a problem.&lt;br&gt;
&lt;br&gt;
I am starting to wonder if its time to&lt;br&gt;
reassess those growdata codes as Matlab&lt;br&gt;
evolves. In some recent tests it seemed&lt;br&gt;
that the problem I had seen with creating&lt;br&gt;
large cell arrays with millions of elements&lt;br&gt;
may no longer be as much of an issue in&lt;br&gt;
recent releases. The underlying matlab&lt;br&gt;
codes must have been improved. I'll want&lt;br&gt;
to compare how use of a structure compares&lt;br&gt;
to a cell array for memory and speed.&lt;br&gt;
&lt;br&gt;
Some quick testing shows a structure and&lt;br&gt;
a cell array to be similar in time and&lt;br&gt;
memory here.&lt;br&gt;
&lt;br&gt;
tic&lt;br&gt;
for i = 1:1000&lt;br&gt;
&amp;nbsp;&amp;nbsp;a(i).X =rand(1,3);&lt;br&gt;
end,&lt;br&gt;
whos a&lt;br&gt;
toc&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&amp;nbsp;Name Size Bytes Class Attributes&lt;br&gt;
&amp;nbsp;&amp;nbsp;a 1x1000 84064 struct&lt;br&gt;
&lt;br&gt;
Elapsed time is 0.023403 seconds.&lt;br&gt;
&lt;br&gt;
tic&lt;br&gt;
for i = 1:1000&lt;br&gt;
&amp;nbsp;&amp;nbsp;b{i} =rand(1,3);&lt;br&gt;
end&lt;br&gt;
whos b&lt;br&gt;
toc&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&amp;nbsp;Name Size Bytes Class Attributes&lt;br&gt;
&amp;nbsp;&amp;nbsp;b 1x1000 84000 cell&lt;br&gt;
&lt;br&gt;
Elapsed time is 0.022017 seconds.&lt;br&gt;
&lt;br&gt;
The question is, do these scale well with&lt;br&gt;
larger numbers of append operations?&lt;br&gt;
&lt;br&gt;
tic&lt;br&gt;
for i = 1:100000&lt;br&gt;
&amp;nbsp;&amp;nbsp;a(i).X =rand(1,3);&lt;br&gt;
end&lt;br&gt;
whos a&lt;br&gt;
toc&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&amp;nbsp;Name Size Bytes Class Attributes&lt;br&gt;
&amp;nbsp;&amp;nbsp;a 1x100000 8400064 struct&lt;br&gt;
&lt;br&gt;
Elapsed time is 118.482410 seconds.&lt;br&gt;
&lt;br&gt;
This has grown by more than 100x. In fact,&lt;br&gt;
it grew in time by a factor of roughly&lt;br&gt;
10000, or 100^2. It is this quadratic&lt;br&gt;
growth in time that I was trying to avoid&lt;br&gt;
with the growdata codes.&lt;br&gt;
&lt;br&gt;
How about growdata2 here?&lt;br&gt;
&lt;br&gt;
tic&lt;br&gt;
g=growdata2;&lt;br&gt;
for i = 1:100000&lt;br&gt;
&amp;nbsp;&amp;nbsp;g(rand(1,3))&lt;br&gt;
end&lt;br&gt;
g=g();&lt;br&gt;
whos g&lt;br&gt;
toc&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&amp;nbsp;Name Size Bytes Class Attributes&lt;br&gt;
&amp;nbsp;&amp;nbsp;g 100000x3 2400000 double&lt;br&gt;
&lt;br&gt;
Elapsed time is 12.672389 seconds.&lt;br&gt;
&lt;br&gt;
So if you are appending only a few thousand&lt;br&gt;
elements, then either the structure or the&lt;br&gt;
cell array will work quite well. If you&lt;br&gt;
expect to append hundreds of thousands or&lt;br&gt;
more chunks, then the hybrid approach in&lt;br&gt;
the growdata tools is still more reasonable.&lt;br&gt;
&lt;br&gt;
Your mileage may vary with a newer release&lt;br&gt;
yet than R2006b. I've not gotten the latest&lt;br&gt;
release installed until I install a new OS&lt;br&gt;
version here.&lt;br&gt;
&lt;br&gt;
John</description>
    </item>
    <item>
      <pubDate>Thu, 24 May 2007 20:35:02 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374821</link>
      <author>NZTideMan</author>
      <description>On May 25, 2:50 pm, &quot;John D'Errico&quot; &amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
wrote:&lt;br&gt;
&amp;gt; NZTideMan wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; On May 25, 12:35 pm, &quot;John D'Errico&quot;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; wrote:&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; I don't know how big will the vector grow each time. The&lt;br&gt;
&amp;gt; space&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; depends on the input data which will change most of the&lt;br&gt;
&amp;gt; time&lt;br&gt;
&amp;gt; &amp;gt; and I&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; have no way of know the final size.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Take a look at my growdata or growdata2&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; codes. They came from discussions here&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; on c.s-s.m, and were written to allow&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; arrays to grow in size with a minimum&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; penalty in time. My goal was to allow&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; arrays to grow in size to potentially&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; millions of elements, without advance&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; knowledge of the final count. These codes&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; use cell arrays, each cell of which is&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; preallocated to use moderately large&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; but still manageable chunks of memory.&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Arrays with huge numbers of cells can&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; have problems too.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&lt;/a&gt;&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Of course, its always better to just&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; preallocate your large arrays whenever&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; you can.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; HTh,&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; John&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; I didn't know about your growdata routines, John, so I have devised&lt;br&gt;
&amp;gt; &amp;gt; the following way for accumulating results of a Monte Carlo&lt;br&gt;
&amp;gt; &amp;gt; simulation:&lt;br&gt;
&amp;gt; &amp;gt; I generate a structure:&lt;br&gt;
&amp;gt; &amp;gt; s(irun).ymx=a;&lt;br&gt;
&amp;gt; &amp;gt; where irun is the Monte Carlo run number, and the result of the&lt;br&gt;
&amp;gt; &amp;gt; simulation a(1,n) is a row vector where n is the number of events&lt;br&gt;
&amp;gt; &amp;gt; (drawn from a Poisson distribution). n varies from run to run.&lt;br&gt;
&amp;gt; &amp;gt; I understand that when you do it this way, each array in s uses&lt;br&gt;
&amp;gt; &amp;gt; contiguous memory, but they are not contiguous within the&lt;br&gt;
&amp;gt; &amp;gt; structure.&lt;br&gt;
&amp;gt; &amp;gt; Thus, the memory allocation problem does not arise.&lt;br&gt;
&amp;gt; &amp;gt; I'd appreciate your comments on this.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; This is also a reassonable way to solve the&lt;br&gt;
&amp;gt; problem.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Use of a single cell for each new appended&lt;br&gt;
&amp;gt; element is another way that is very similar&lt;br&gt;
&amp;gt; to the structure approach that you used.&lt;br&gt;
&amp;gt; Use of pointers here is more efficient&lt;br&gt;
&amp;gt; than growing and re-allocating the entire&lt;br&gt;
&amp;gt; array. I'm not sure about use of structures,&lt;br&gt;
&amp;gt; as I thought that structures were not&lt;br&gt;
&amp;gt; efficient users of memory as compared to&lt;br&gt;
&amp;gt; cell arrays. This is probably changing&lt;br&gt;
&amp;gt; with each matlab release of course. (See&lt;br&gt;
&amp;gt; below for a comparison.)&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; An issue is how many total elements that&lt;br&gt;
&amp;gt; you may expect to see. In some earlier&lt;br&gt;
&amp;gt; testing, I found that cell arrays with&lt;br&gt;
&amp;gt; millions of cells caused some serious&lt;br&gt;
&amp;gt; problems, however a few thousand, or even&lt;br&gt;
&amp;gt; ten thousand or so cells was not a problem.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; I am starting to wonder if its time to&lt;br&gt;
&amp;gt; reassess those growdata codes as Matlab&lt;br&gt;
&amp;gt; evolves. In some recent tests it seemed&lt;br&gt;
&amp;gt; that the problem I had seen with creating&lt;br&gt;
&amp;gt; large cell arrays with millions of elements&lt;br&gt;
&amp;gt; may no longer be as much of an issue in&lt;br&gt;
&amp;gt; recent releases. The underlying matlab&lt;br&gt;
&amp;gt; codes must have been improved. I'll want&lt;br&gt;
&amp;gt; to compare how use of a structure compares&lt;br&gt;
&amp;gt; to a cell array for memory and speed.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Some quick testing shows a structure and&lt;br&gt;
&amp;gt; a cell array to be similar in time and&lt;br&gt;
&amp;gt; memory here.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;   a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end,&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   a 1x1000 84064 struct&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 0.023403 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;   b{i} =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos b&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   b 1x1000 84000 cell&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 0.022017 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; The question is, do these scale well with&lt;br&gt;
&amp;gt; larger numbers of append operations?&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;   a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   a 1x100000 8400064 struct&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 118.482410 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; This has grown by more than 100x. In fact,&lt;br&gt;
&amp;gt; it grew in time by a factor of roughly&lt;br&gt;
&amp;gt; 10000, or 100^2. It is this quadratic&lt;br&gt;
&amp;gt; growth in time that I was trying to avoid&lt;br&gt;
&amp;gt; with the growdata codes.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; How about growdata2 here?&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; g=growdata2;&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;   g(rand(1,3))&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; g=g();&lt;br&gt;
&amp;gt; whos g&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   g 100000x3 2400000 double&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 12.672389 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; So if you are appending only a few thousand&lt;br&gt;
&amp;gt; elements, then either the structure or the&lt;br&gt;
&amp;gt; cell array will work quite well. If you&lt;br&gt;
&amp;gt; expect to append hundreds of thousands or&lt;br&gt;
&amp;gt; more chunks, then the hybrid approach in&lt;br&gt;
&amp;gt; the growdata tools is still more reasonable.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Your mileage may vary with a newer release&lt;br&gt;
&amp;gt; yet than R2006b. I've not gotten the latest&lt;br&gt;
&amp;gt; release installed until I install a new OS&lt;br&gt;
&amp;gt; version here.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John- Hide quoted text -&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; - Show quoted text -&lt;br&gt;
&lt;br&gt;
That's pretty convincing!&lt;br&gt;
I guess the lesson is that if you have toy problems, using a structure&lt;br&gt;
works fine, but for real-world problems growdata2 is the way to go.&lt;br&gt;
But an advantage of the structure approach is that at the end I have&lt;br&gt;
all the runs separated.&lt;br&gt;
Of course, I can concatenate them if I choose using square brackets.</description>
    </item>
    <item>
      <pubDate>Fri, 25 May 2007 10:05:05 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374916</link>
      <author>Eric Sampson</author>
      <description>John D'Errico wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
(snip)&lt;br&gt;
&amp;gt; Some quick testing shows a structure and&lt;br&gt;
&amp;gt; a cell array to be similar in time and&lt;br&gt;
&amp;gt; memory here.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt; a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end,&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt; a 1x1000 84064 struct&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 0.023403 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt; b{i} =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos b&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt; b 1x1000 84000 cell&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 0.022017 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; The question is, do these scale well with&lt;br&gt;
&amp;gt; larger numbers of append operations?&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt; a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt; a 1x100000 8400064 struct&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 118.482410 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; This has grown by more than 100x. In fact,&lt;br&gt;
&amp;gt; it grew in time by a factor of roughly&lt;br&gt;
&amp;gt; 10000, or 100^2. It is this quadratic&lt;br&gt;
&amp;gt; growth in time that I was trying to avoid&lt;br&gt;
&amp;gt; with the growdata codes.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; How about growdata2 here?&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; g=growdata2;&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt; g(rand(1,3))&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; g=g();&lt;br&gt;
&amp;gt; whos g&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt; g 100000x3 2400000 double&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 12.672389 seconds.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; So if you are appending only a few thousand&lt;br&gt;
&amp;gt; elements, then either the structure or the&lt;br&gt;
&amp;gt; cell array will work quite well. If you&lt;br&gt;
&amp;gt; expect to append hundreds of thousands or&lt;br&gt;
&amp;gt; more chunks, then the hybrid approach in&lt;br&gt;
&amp;gt; the growdata tools is still more reasonable.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Your mileage may vary with a newer release&lt;br&gt;
&amp;gt; yet than R2006b. I've not gotten the latest&lt;br&gt;
&amp;gt; release installed until I install a new OS&lt;br&gt;
&amp;gt; version here.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John&lt;br&gt;
&amp;nbsp;&amp;nbsp;&lt;br&gt;
&lt;br&gt;
John, I can't reproduce your results on my computer (winxp32), on&lt;br&gt;
R2006b or R2007a. The cell and struct both take virtually the same&lt;br&gt;
amount of time, about 20sec for the 100000 case. Growdata is still&lt;br&gt;
faster, of course, coming in at about 10sec.&lt;br&gt;
&lt;br&gt;
Regards,&lt;br&gt;
Eric Sampson&lt;br&gt;
The MathWorks, Inc.</description>
    </item>
    <item>
      <pubDate>Fri, 25 May 2007 10:35:59 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#374921</link>
      <author>John D'Errico</author>
      <description>Eric Sampson wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John D'Errico wrote:&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; (snip)&lt;br&gt;
&amp;gt;&amp;gt; Some quick testing shows a structure and&lt;br&gt;
&amp;gt;&amp;gt; a cell array to be similar in time and&lt;br&gt;
&amp;gt;&amp;gt; memory here.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; tic&lt;br&gt;
&amp;gt;&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;&amp;gt; a(i).X =rand(1,3);&lt;br&gt;
&amp;gt;&amp;gt; end,&lt;br&gt;
&amp;gt;&amp;gt; whos a&lt;br&gt;
&amp;gt;&amp;gt; toc&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;&amp;gt; a 1x1000 84064 struct&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Elapsed time is 0.023403 seconds.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; tic&lt;br&gt;
&amp;gt;&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;&amp;gt; b{i} =rand(1,3);&lt;br&gt;
&amp;gt;&amp;gt; end&lt;br&gt;
&amp;gt;&amp;gt; whos b&lt;br&gt;
&amp;gt;&amp;gt; toc&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;&amp;gt; b 1x1000 84000 cell&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Elapsed time is 0.022017 seconds.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; The question is, do these scale well with&lt;br&gt;
&amp;gt;&amp;gt; larger numbers of append operations?&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; tic&lt;br&gt;
&amp;gt;&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;&amp;gt; a(i).X =rand(1,3);&lt;br&gt;
&amp;gt;&amp;gt; end&lt;br&gt;
&amp;gt;&amp;gt; whos a&lt;br&gt;
&amp;gt;&amp;gt; toc&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;&amp;gt; a 1x100000 8400064 struct&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Elapsed time is 118.482410 seconds.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; This has grown by more than 100x. In fact,&lt;br&gt;
&amp;gt;&amp;gt; it grew in time by a factor of roughly&lt;br&gt;
&amp;gt;&amp;gt; 10000, or 100^2. It is this quadratic&lt;br&gt;
&amp;gt;&amp;gt; growth in time that I was trying to avoid&lt;br&gt;
&amp;gt;&amp;gt; with the growdata codes.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; How about growdata2 here?&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; tic&lt;br&gt;
&amp;gt;&amp;gt; g=growdata2;&lt;br&gt;
&amp;gt;&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;&amp;gt; g(rand(1,3))&lt;br&gt;
&amp;gt;&amp;gt; end&lt;br&gt;
&amp;gt;&amp;gt; g=g();&lt;br&gt;
&amp;gt;&amp;gt; whos g&lt;br&gt;
&amp;gt;&amp;gt; toc&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;&amp;gt; g 100000x3 2400000 double&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Elapsed time is 12.672389 seconds.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; So if you are appending only a few thousand&lt;br&gt;
&amp;gt;&amp;gt; elements, then either the structure or the&lt;br&gt;
&amp;gt;&amp;gt; cell array will work quite well. If you&lt;br&gt;
&amp;gt;&amp;gt; expect to append hundreds of thousands or&lt;br&gt;
&amp;gt;&amp;gt; more chunks, then the hybrid approach in&lt;br&gt;
&amp;gt;&amp;gt; the growdata tools is still more reasonable.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Your mileage may vary with a newer release&lt;br&gt;
&amp;gt;&amp;gt; yet than R2006b. I've not gotten the latest&lt;br&gt;
&amp;gt;&amp;gt; release installed until I install a new OS&lt;br&gt;
&amp;gt;&amp;gt; version here.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; John&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John, I can't reproduce your results on my computer (winxp32), on&lt;br&gt;
&amp;gt; R2006b or R2007a. The cell and struct both take virtually the same&lt;br&gt;
&amp;gt; amount of time, about 20sec for the 100000 case. Growdata is still&lt;br&gt;
&amp;gt; faster, of course, coming in at about 10sec.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Regards,&lt;br&gt;
&amp;gt; Eric Sampson&lt;br&gt;
&amp;gt; The MathWorks, Inc.&lt;br&gt;
&amp;nbsp;&amp;nbsp;&lt;br&gt;
The cell and the struct were both the same&lt;br&gt;
for me, but it was definitely 2 minutes.&lt;br&gt;
&lt;br&gt;
Its possible this is a CPU issue, since&lt;br&gt;
I'm running on a Mac, or a memory limitation&lt;br&gt;
issue (3/4 gig of ram.) I was running a web&lt;br&gt;
browser on the side, so there might have&lt;br&gt;
been a virtual problem.&lt;br&gt;
&lt;br&gt;
I'll re-run the test this afternoon with&lt;br&gt;
as little else competing for ram as possible&lt;br&gt;
(after I also reboot.)&lt;br&gt;
&lt;br&gt;
John</description>
    </item>
    <item>
      <pubDate>Fri, 25 May 2007 16:24:01 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#375001</link>
      <author>John D'Errico</author>
      <description>John D'Errico wrote:&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; John, I can't reproduce your results on my computer (winxp32),&lt;br&gt;
on&lt;br&gt;
&amp;gt;&amp;gt; R2006b or R2007a. The cell and struct both take virtually the&lt;br&gt;
&amp;gt; same&lt;br&gt;
&amp;gt;&amp;gt; amount of time, about 20sec for the 100000 case. Growdata is&lt;br&gt;
&amp;gt; still&lt;br&gt;
&amp;gt;&amp;gt; faster, of course, coming in at about 10sec.&lt;br&gt;
&amp;gt;&amp;gt;&lt;br&gt;
&amp;gt;&amp;gt; Regards,&lt;br&gt;
&amp;gt;&amp;gt; Eric Sampson&lt;br&gt;
&amp;gt;&amp;gt; The MathWorks, Inc.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; The cell and the struct were both the same&lt;br&gt;
&amp;gt; for me, but it was definitely 2 minutes.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Its possible this is a CPU issue, since&lt;br&gt;
&amp;gt; I'm running on a Mac, or a memory limitation&lt;br&gt;
&amp;gt; issue (3/4 gig of ram.) I was running a web&lt;br&gt;
&amp;gt; browser on the side, so there might have&lt;br&gt;
&amp;gt; been a virtual problem.&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; I'll re-run the test this afternoon with&lt;br&gt;
&amp;gt; as little else competing for ram as possible&lt;br&gt;
&amp;gt; (after I also reboot.)&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John&lt;br&gt;
&amp;nbsp;&amp;nbsp;&lt;br&gt;
I reran my test. Both the struct and the&lt;br&gt;
cell take 100+ seconds on my machine, not&lt;br&gt;
touching the mouse during that time.&lt;br&gt;
&lt;br&gt;
tic,for i = 1:100000,b{i} =rand(1,3);end,toc&lt;br&gt;
&lt;br&gt;
Elapsed time is 100.453774 seconds.&lt;br&gt;
&lt;br&gt;
I did leave a cpu monitor running, and it&lt;br&gt;
showed high cpu utilization for the entire&lt;br&gt;
time. There was no hard disk thrashing to&lt;br&gt;
go to virtual memory.&lt;br&gt;
&lt;br&gt;
The growdata2 call was consistently about&lt;br&gt;
12 seconds.&lt;br&gt;
&lt;br&gt;
John</description>
    </item>
    <item>
      <pubDate>Fri, 25 May 2007 17:04:00 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#375010</link>
      <author>us</author>
      <description>John D'Errico:&lt;br&gt;
&amp;lt;SNIP growing up fast...&lt;br&gt;
&lt;br&gt;
here are my values for &amp;lt;john d'errico&amp;gt;'s test suite&lt;br&gt;
- n = 100000&lt;br&gt;
- 1) struct&lt;br&gt;
- 2) cell&lt;br&gt;
- 3) growdata2&lt;br&gt;
- wintel env: ic2d/2*2.4ghz/2gb/winvista/r2007a&lt;br&gt;
- reboot, ML only&lt;br&gt;
&lt;br&gt;
Elapsed time is 6.428093 seconds. &amp;lt;- struct&lt;br&gt;
Elapsed time is 6.187052 seconds. &amp;lt;- cell&lt;br&gt;
Elapsed time is 2.578906 seconds. &amp;lt;- growdata2&lt;br&gt;
&lt;br&gt;
us</description>
    </item>
    <item>
      <pubDate>Fri, 25 May 2007 21:06:34 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vecto</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#375041</link>
      <author>John D'Errico</author>
      <description>us wrote:&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; John D'Errico:&lt;br&gt;
&amp;gt; &amp;lt;SNIP growing up fast...&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; here are my values for &amp;lt;john d'errico&amp;gt;'s test suite&lt;br&gt;
&amp;gt; - n = 100000&lt;br&gt;
&amp;gt; - 1) struct&lt;br&gt;
&amp;gt; - 2) cell&lt;br&gt;
&amp;gt; - 3) growdata2&lt;br&gt;
&amp;gt; - wintel env: ic2d/2*2.4ghz/2gb/winvista/r2007a&lt;br&gt;
&amp;gt; - reboot, ML only&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; Elapsed time is 6.428093 seconds. &amp;lt;- struct&lt;br&gt;
&amp;gt; Elapsed time is 6.187052 seconds. &amp;lt;- cell&lt;br&gt;
&amp;gt; Elapsed time is 2.578906 seconds. &amp;lt;- growdata2&lt;br&gt;
&amp;gt;&lt;br&gt;
&amp;gt; us&lt;br&gt;
&amp;nbsp;&amp;nbsp;&lt;br&gt;
Gotta get me one of them souped up computers.&lt;br&gt;
&lt;br&gt;
Of more interest is why is there a 10:1&lt;br&gt;
ratio on my system?&lt;br&gt;
&lt;br&gt;
John</description>
    </item>
    <item>
      <pubDate>Mon, 28 May 2007 01:23:09 -0400</pubDate>
      <title>Re: decrease in speed due to appending a row vector</title>
      <link>http://www.mathworks.com/matlabcentral/newsreader/view_thread/148947#375229</link>
      <author>Loren Shure</author>
      <description>In article &amp;lt;ef580c1.5@webcrossing.raydaftYaTP&amp;gt;, &lt;br&gt;
woodchips@rochester.rr.com says...&lt;br&gt;
&amp;gt; NZTideMan wrote:&lt;br&gt;
&amp;gt; &amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; On May 25, 12:35 pm, &quot;John D'Errico&quot;&lt;br&gt;
&amp;gt; &amp;lt;woodch...@rochester.rr.com&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; wrote:&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Brian wrote:&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; I don't know how big will the vector grow each time. The&lt;br&gt;
&amp;gt; space&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; depends on the input data which will change most of the&lt;br&gt;
&amp;gt; time&lt;br&gt;
&amp;gt; &amp;gt; and I&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;gt; have no way of know the final size.&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Take a look at my growdata or growdata2&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; codes. They came from discussions here&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; on c.s-s.m, and were written to allow&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; arrays to grow in size with a minimum&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; penalty in time. My goal was to allow&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; arrays to grow in size to potentially&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; millions of elements, without advance&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; knowledge of the final count. These codes&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; use cell arrays, each cell of which is&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; preallocated to use moderately large&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; but still manageable chunks of memory.&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Arrays with huge numbers of cells can&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; have problems too.&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; &amp;lt;&lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&quot;&gt;http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objec...&lt;/a&gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; Of course, its always better to just&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; preallocate your large arrays whenever&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; you can.&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt;&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; HTh,&lt;br&gt;
&amp;gt; &amp;gt;&amp;gt; John&lt;br&gt;
&amp;gt; &amp;gt;&lt;br&gt;
&amp;gt; &amp;gt; I didn't know about your growdata routines, John, so I have devised&lt;br&gt;
&amp;gt; &amp;gt; the following way for accumulating results of a Monte Carlo&lt;br&gt;
&amp;gt; &amp;gt; simulation:&lt;br&gt;
&amp;gt; &amp;gt; I generate a structure:&lt;br&gt;
&amp;gt; &amp;gt; s(irun).ymx=a;&lt;br&gt;
&amp;gt; &amp;gt; where irun is the Monte Carlo run number, and the result of the&lt;br&gt;
&amp;gt; &amp;gt; simulation a(1,n) is a row vector where n is the number of events&lt;br&gt;
&amp;gt; &amp;gt; (drawn from a Poisson distribution). n varies from run to run.&lt;br&gt;
&amp;gt; &amp;gt; I understand that when you do it this way, each array in s uses&lt;br&gt;
&amp;gt; &amp;gt; contiguous memory, but they are not contiguous within the&lt;br&gt;
&amp;gt; &amp;gt; structure.&lt;br&gt;
&amp;gt; &amp;gt; Thus, the memory allocation problem does not arise.&lt;br&gt;
&amp;gt; &amp;gt; I'd appreciate your comments on this.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; This is also a reassonable way to solve the&lt;br&gt;
&amp;gt; problem.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Use of a single cell for each new appended&lt;br&gt;
&amp;gt; element is another way that is very similar&lt;br&gt;
&amp;gt; to the structure approach that you used.&lt;br&gt;
&amp;gt; Use of pointers here is more efficient&lt;br&gt;
&amp;gt; than growing and re-allocating the entire&lt;br&gt;
&amp;gt; array. I'm not sure about use of structures,&lt;br&gt;
&amp;gt; as I thought that structures were not&lt;br&gt;
&amp;gt; efficient users of memory as compared to&lt;br&gt;
&amp;gt; cell arrays. This is probably changing&lt;br&gt;
&amp;gt; with each matlab release of course. (See&lt;br&gt;
&amp;gt; below for a comparison.)&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; An issue is how many total elements that&lt;br&gt;
&amp;gt; you may expect to see. In some earlier&lt;br&gt;
&amp;gt; testing, I found that cell arrays with&lt;br&gt;
&amp;gt; millions of cells caused some serious&lt;br&gt;
&amp;gt; problems, however a few thousand, or even&lt;br&gt;
&amp;gt; ten thousand or so cells was not a problem.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; I am starting to wonder if its time to&lt;br&gt;
&amp;gt; reassess those growdata codes as Matlab&lt;br&gt;
&amp;gt; evolves. In some recent tests it seemed&lt;br&gt;
&amp;gt; that the problem I had seen with creating&lt;br&gt;
&amp;gt; large cell arrays with millions of elements&lt;br&gt;
&amp;gt; may no longer be as much of an issue in&lt;br&gt;
&amp;gt; recent releases. The underlying matlab&lt;br&gt;
&amp;gt; codes must have been improved. I'll want&lt;br&gt;
&amp;gt; to compare how use of a structure compares&lt;br&gt;
&amp;gt; to a cell array for memory and speed.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Some quick testing shows a structure and&lt;br&gt;
&amp;gt; a cell array to be similar in time and&lt;br&gt;
&amp;gt; memory here.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;   a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end,&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   a 1x1000 84064 struct&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Elapsed time is 0.023403 seconds.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:1000&lt;br&gt;
&amp;gt;   b{i} =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos b&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   b 1x1000 84000 cell&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Elapsed time is 0.022017 seconds.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; The question is, do these scale well with&lt;br&gt;
&amp;gt; larger numbers of append operations?&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;   a(i).X =rand(1,3);&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; whos a&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   a 1x100000 8400064 struct&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Elapsed time is 118.482410 seconds.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; This has grown by more than 100x. In fact,&lt;br&gt;
&amp;gt; it grew in time by a factor of roughly&lt;br&gt;
&amp;gt; 10000, or 100^2. It is this quadratic&lt;br&gt;
&amp;gt; growth in time that I was trying to avoid&lt;br&gt;
&amp;gt; with the growdata codes.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; How about growdata2 here?&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; tic&lt;br&gt;
&amp;gt; g=growdata2;&lt;br&gt;
&amp;gt; for i = 1:100000&lt;br&gt;
&amp;gt;   g(rand(1,3))&lt;br&gt;
&amp;gt; end&lt;br&gt;
&amp;gt; g=g();&lt;br&gt;
&amp;gt; whos g&lt;br&gt;
&amp;gt; toc&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt;   Name Size Bytes Class Attributes&lt;br&gt;
&amp;gt;   g 100000x3 2400000 double&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Elapsed time is 12.672389 seconds.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; So if you are appending only a few thousand&lt;br&gt;
&amp;gt; elements, then either the structure or the&lt;br&gt;
&amp;gt; cell array will work quite well. If you&lt;br&gt;
&amp;gt; expect to append hundreds of thousands or&lt;br&gt;
&amp;gt; more chunks, then the hybrid approach in&lt;br&gt;
&amp;gt; the growdata tools is still more reasonable.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; Your mileage may vary with a newer release&lt;br&gt;
&amp;gt; yet than R2006b. I've not gotten the latest&lt;br&gt;
&amp;gt; release installed until I install a new OS&lt;br&gt;
&amp;gt; version here.&lt;br&gt;
&amp;gt; &lt;br&gt;
&amp;gt; John&lt;br&gt;
&amp;gt; &lt;br&gt;
&lt;br&gt;
structs and cells should generally perform about the same.  Depends on &lt;br&gt;
what you are doing (accessing, appending, overwriting), where you use &lt;br&gt;
them (command line, function, script), but structs have a slight to &lt;br&gt;
larger advantage sometimes.&lt;br&gt;
&lt;br&gt;
-- Loren&lt;br&gt;
&lt;a href=&quot;http://blogs.mathworks.com/loren/&quot;&gt;http://blogs.mathworks.com/loren/&lt;/a&gt;</description>
    </item>
  </channel>
</rss>

