Creating Structure Names Dynamically

Hello,

For those willing enough to help, please read my ENTIRE post to understand my "unique" problem before posting an answer!

I would like to dynamically create structure NAMES (not fields.) Please continue reading! I've already read and done research on this topic, and it seems that doing this is not good programming. However, the problem is that I'm dealing with large amounts of files and data that need to be split up and processed. Here's my question:

We have the following variable integers:

N = 1:infinity (theoretically of course, realistically maybe 1:30)

X = 1:10

Y = 1:14

I need to individually create the following structures, either manually or dynamically in a loop (hopefully!!!):

BiasStats_Dataset1.BandX.FPMY.Value = …

BiasStats_Dataset2.BandX.FPMY.Value = …

BiasStats_Dataset3.BandX.FPMY.Value = …

BiasStats_DatasetN.BandX.FPMY.Value = …

.

.

CStats_Dataset1.BandX.FPMY.Value = …

CStats_Dataset2.BandX.FPMY.Value = …

CStats_Dataset3.BandX.FPMY.Value = …

CStats_DatasetN.BandX.FPMY.Value = …

.

.

… and so on.

So as you can see, I will be creating many individual structures to decrease file sizes. The way I’m currently doing it is:

BiasStats.Dataset1.BandX.FPMY.Value = …

BiasStats.Dataset2.BandX.FPMY.Value = …

BiasStats.DatasetN.BandX.FPMY.Value = …

.

.

CStats.Dataset1.BandX.FPMY.Value = …

CStats.Dataset2.BandX.FPMY.Value = …

CStats.DatasetN.BandX.FPMY.Value = …

.

.

… and so on.

Doing it my way now, I will end up with a total of 6 individual structures (BiasStats and CStats are two of them.) This way works very well from an organizational standpoint, but each structure (BiasStats for example) takes up over 2GB of memory when the processing is finished! And I’m only working with 3 datasets at the moment (N = 3). This is where my problem lies. My code runs fine on our server because it has 24 GB of memory, but I feel I increase the number of datasets(N) to maybe 20 or 30, even our server might run out of memory because the structures are continually growing with each new dataset.

I will continue researching, and working with the ‘eval’ function, as that seems to be a possible way to go about this. I think it will be necessary for me in the future to create many individual structures for each dataset_contents, and save/load them from the hard drive rather than storing them in memory. Either that, or use an ENTIRELY different approach to this problem. Either way, data organization and dynamic accessibility to the data (with loops) is absolutely key. I'm working with far too much data to manually access and create structures by hand.

Thanks to anyone willing to take the time to help me on this, much appreciated!!!

Answer 1

[Sean de Wolski]

Don't do it!

FAQ4.6

[Brad]

How would you propose I use an array to do what I'm trying to do?

[Sean de Wolski]

Use a cell array

Answer 2

[Walter Roberson]

BiasStats.Dataset(N).Band(X).FPM(Y).Value

But beyond that, to give you suggestions on how the data might be used effectively without filling memory, you will need to give us an idea of how you are going to use the data. Random access, or along a particular dimension? Will the access be read-only later or will it be read/write ?

[Brad]

"BiasStats.Dataset(N).Band(X).FPM(Y).Value" ---

That is exactly how I am currently organizing my structures. This works very well from an organizational standpoint. The problem is, the "BiasStats" structure alone is over 2 GB, and there are 5 other structures that same size. Too big! I need to break down my structures further to decrease the size of them, and save/load them from the hard drive into memory.

"But beyond that..." ---

So, each dataset is essentially a very large "image" file. In each image file, there are 9 spectral bands (actually 10, one is 'blind'.) There are also 14 FPMs (focal plane modules), each one containing 1/14th of the image data from a specific band. I am currently working with 6 separate image/data files, one 'dark' and one 'light' (or one dataset, is what I refer to a dark and a light, although there are actually two.) However, there are HUNDREDS of other images to be processed.

In a nutshell, I read-in the datasets using a function (looped), set the band and fpm in a loop, do a bunch of processing, creating structures, saving data, computing statistics, saving statistics, filtering these statistics, saving them in a structure, etc etc. In the end, I come out with 6 separate structures (BiasStats, SStats, BCStats, CStats, RGStats, FilterStats), each one over 2 GB in size. If I increase the number of datasets I am using to say 50 datasets, one structure could end up being 30 GB or more... much too large to be stored in memory.

Ideally, I would like to break down my structures further, to decrease the file size of them, and then save/load them from HDD to memory. It's obviously slower accessing the HDD, but with so much data, it's the only way I believe. I would like to break down my structures as such:

BiasStats_Dataset1.BandX.FPMY.Value = …

BiasStats_Dataset2.BandX.FPMY.Value = …

BiasStats_Dataset3.BandX.FPMY.Value = …

.

.

.

BiasStats_DatasetN.BandX.FPMY.Value = …

and so on for the other structures...

CStats_DatasetN.BandX.FPMY.Value = …

BCStats_DatasetN.BandX.FPMY.Value = …

FilterStats_Dataset1N.BandX.FPMY.Value = …

SStats_Dataset1N.BandX.FPMY.Value = …

RGStats_Dataset1N.BandX.FPMY.Value = …

Does this help clarify?

Answer 3

[Brad]

Along with the help of a co-worker, I have solved the issue with the following code!

%%%% Dynamically Create Structures %%%%%

for n = 1:20; % number of datasets

      for band = 1:10; % bands 1-10

          for fpm = 1:14; % fpms 1-14
              eval(['BiasStats_Dataset',num2str(n),'.Band' num2str(band) '.FPM' num2str(fpm) '.StatisticValue=0']);
          end;
      end;
  end;

This solves the creation issue. Next is how to efficiently access them. I believe the only way to access them in a loop will be to continue using the 'eval' function. I can see this getting clumsy, but, if it works... then great.

[David Young]

You'd be a great deal better off using cell arrays. You're right that this is clumsy, and anything you can do like this you can do more efficiently with an array.