How to search for a index to a variable with known size

Hi All

I'm still relative new to MATLAB and I'm struggeling a bit to find an index. Following info I have received from a 'whos' command.

Now I want to 'find' the entry 'simTimeData' by searching the array based on the size, so something like:

index = find({variableList.size}==[226411,1])
index = 12

For me the size of the variable will be known, but it's name can change. Resulting furthermore in an unknown position in the array. That is why I want to 'search' it based on the size. Thanks for any suggestion.

With kind regards

Ludo Houben

[Stephen23]

So far you have not told us the most important information: how did you get those variables into the workspace?

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

[Ludo Houben]

Hi Stephen23

Thank you for your reply, but I see this reply to almost every message that I have been searching the last days. I'm not asking about using the 'eval' function. I'm not talking about a FOR LOOP that is generating Var1, Var2, VarN variables. So I think that your answer is in this situation not contributing.

But to answer your question. I get the info from *.mat / *.m and *.csv files. The function that I'm building is gathering the info from the files and returns the data I'm interested in, and transfers it to 'known' variables. In more details, a PLC generates binary LOG data which is random from a point of view in MATLAB. There is a tool that reads the STRUCT files from the PLC and converts the binary data into the *.csv file. The tool is also creating a MATLAB *.m script that can convert the *.csv data into a MATLAB structure. However the user can freely choose the name of the main variable to generate. This is the point that I cannot control and I'm trying to find a solution for. However the user can also have saved previous 'manual' created *.mat files for the model to run and I want to function to coop also with this information.

For the future I will be implementing the external tool into MATLAB with fixed variable names, but there is no time for implementing this at this time. So the current 'solution' is practical for now in the proces of further automating a fully manual process up to a year ago.

With kind regards

Ludo

[Stephen23]

"Thank you for your reply, but I see this reply to almost every message that I have been searching the last days."

That should be a big hint.

"I'm not asking about using the 'eval' function."

That is good, because if you think that my tutorial is about the EVAL function, then you have misunderstood it.

"I'm not talking about a FOR LOOP that is generating Var1, Var2, VarN variables."

That is also good, because my tutorial is not about "generating... variables" but is about accessing variable names dynamically.

"But to answer your question. I get the info from *.mat / *.m and *.csv files."

.MAT files you can always LOAD into an output variable and then acess its fields. This is explained in my tutorial.

For importing any other filetypes you can use whatever variable name you want, so should not be a problem.

"The tool is also creating a MATLAB *.m script that can convert the *.csv data into a MATLAB structure. However the user can freely choose the name of the main variable to generate. This is the point that I cannot control and I'm trying to find a solution for."

I agree that is most unfortunate data design. Sadly sometimes one has to work with such things.

Do you RUN that script in your code? Does it start with the FUNCTION keyword?

Note the difference: https://www.mathworks.com/help/matlab/matlab_prog/scripts-and-functions.html

Once you answer that, then most likely we can think about better ways to approach this task.

"However the user can also have saved previous 'manual' created *.mat files for the model to run and I want to function to coop also with this information."

I don't see why that should be a problem, just LOAD into an output variable and access its fields:

https://www.mathworks.com/help/matlab/ref/fieldnames.html

https://www.mathworks.com/help/matlab/matlab_prog/generate-field-names-from-variables.html

I appreciate your explanation. Unlike others I will not just answer your question:

https://xyproblem.info/

https://en.wikipedia.org/wiki/XY_problem

but instead try to understand what is the actual goal and how it can be achieved. As the MATLAB documentation clearly states "Avoid functions that query the state of MATLAB such as inputname, which, whos, exist(var), and dbstack. Run-time introspection is computationally expensive."

Source: https://www.mathworks.com/help/matlab/matlab_prog/techniques-for-improving-performance.html

If you are interested and want to learn more about how to write better MATLAB code then I am most happy to help you. Struggling with variable names or other kinds of programming introspection (e.g. relying on WHOS) is often solved very easily.

[Ludo Houben]

Hi Stephen23

I'm a 20+ years experienced PLC programmer who has just started using MATLAB. The others that worked with MATLAB before in the past were people that just have left school and did not have any guidance, so they 'did' something. I'm now trying to create/introduce a more structured way of working.

In general we have a simulink model which runs in a PLC (Beckhoff). We feed stored logging data into the simulink model to verify its behaviour. The functions that I'm currently working on shall support the loading of the model and preparing the base workspace for running the model. I want the base workspace be able to run with a default naming for data and time information, which will be derived from the logged data.

Each PLC scan, the data is logged into a file in binairy format. So the data must be extracted from the file and put into a structure to be able to use it in MATLAB for feeding the simulink model. The structure is known in the PLC and is buildup from multiple blocks (IOOP Industrial Object Oriented Programming). So the external tool harvest the data, necessary to create the struct in MATLAB, from a buch of separate PLC (CoDeSys) files. In the past this struct was created manually by the engineers. This was always a problem, as there was also development in the PLC code.

The external tool supplies a CSV file containing the simulation data and a MATLAB script that will create the complete structure (including data types) and will be putting the chuncks of data into it. This data is used by the model as an input for simulation. In the passt the engineers saved ALL variables into a *.mat file that they created manually. So it contained double data and multiple unused variables that can be cleaned out.

I'm now trying to 'clean' this part up by reading all messy data files and transform them into known variables that shall be used in all future simulink models. With time I will add the data extracting to MATLAB, but at this point in time this is too much effort for the time I have for development. As for now, the performance is not yet an issue, but I will have a look at your suggestions to learn more about it.

Cheers

Ludo

[Ludo Houben]

".MAT files you can always LOAD into an output variable and then acess its fields. This is explained in my tutorial."

Thats correct loading is working, but what variable to look for if you don't know the name? The one thing that I can rely on is that I need 'the biggest' variable. Both the marked files are the ones I need to load. However the 'DetectionLog' is the flexible name comming from an external tool.

"For importing any other filetypes you can use whatever variable name you want, so should not be a problem."

Correct, but still the same issue of not knowing how the variable is called in the imported data.

"I agree that is most unfortunate data design. Sadly sometimes one has to work with such things."

The external tool was a big improvement at its time. Currently I'm creating a new framework for testing and that the MATLAB/simulink functionality will be converted into a (more) OOP structure. The next step will be creating the external tool functionality into MATLAB.

"Do you RUN that script in your code? Does it start with the FUNCTION keyword?

Note the difference: https://www.mathworks.com/help/matlab/matlab_prog/scripts-and-functions.html

Once you answer that, then most likely we can think about better ways to approach this task."

Here is an example of the 'generated' scipt. 'MeasData', 'Time' and 'kk' are fixed namings which I can use when importing directly in matlab. The 'DetectionLog' is the part that the user can change in the external tool. I don't know the naming when a user is loading these scripts and/or *.mat files created with these scripts.

"I don't see why that should be a problem, just LOAD into an output variable and access its fields:

https://www.mathworks.com/help/matlab/ref/fieldnames.html

https://www.mathworks.com/help/matlab/matlab_prog/generate-field-names-from-variables.html"

The fieldname is the part that I'm trying to discover :-)

"I appreciate your explanation. Unlike others I will not just answer your question:

https://xyproblem.info/

https://en.wikipedia.org/wiki/XY_problem

but instead try to understand what is the actual goal and how it can be achieved. As the MATLAB documentation clearly states "Avoid functions that query the state of MATLAB such as inputname, which, whos, exist(var), and dbstack. Run-time introspection is computationally expensive."

Source: https://www.mathworks.com/help/matlab/matlab_prog/techniques-for-improving-performance.html"

Very good (didn't know this XY problem site though). In my case I do not like the current way of working of my fellow (mechatronic engineers) coworkers. So I'm trying to cleanup the 'doing by hand' and 'always using different naming' approach of my colleages. But changes take time (they need to support the changes too).

"If you are interested and want to learn more about how to write better MATLAB code then I am most happy to help you. Struggling with variable names or other kinds of programming introspection (e.g. relying on WHOS) is often solved very easily."

For me it's a lot of learning by doing at this time. As there are no experienced MATLAB programmers in our company (I'm working on that). So I would appreciate this. I already had several questions that I have find out myself. I always try to add my solutions to them and explain it, as other could learn from them as well.

Cheers

Ludo

[Stephen23]

"Thats correct loading is working, but what variable to look for if you don't know the name?"

Once you LOAD into an output variable like this:

S = load(..);

then you can trivially use a loop (using FIELDNAMES and dynamic fieldnames) or STRUCTFUN (or CELLFUN after converting to cell array, etc).

Lets try it right now.

% create fake data in MAT file:
a = rand(2,3,4);
x = 4;
simTimeData = zeros(226411,1);
save test.mat
clearvars
% LOAD data:
S = load('test.mat');
% use STRUCTFUN to select the array with a particular size:
V = [226411,1]; % required size
F = @(a) isequal(size(a),V); % @(a) isscalar(a) && isstruct(a);
X = structfun(F,S);
C = struct2cell(S);
sdt = C{X}

sdt = 226411×1

0 0 0 0 0 0 0 0 0 0

And there is the desired array, without knowing anything about its name and without messing around with ugly introspective coding (WHOS, EVAL, etc). It also makes it much easier to add code checking (which you should be doing, e.g. checking how many arrays match the requested conditions, etc).

If you really want to know the array's name then use FIELDNAMES and the index X.

"Correct, but still the same issue of not knowing how the variable is called in the imported data. "

This statement I do not understand. When importing from e.g. CSV file you always use your variable names in your code.

ThisVariableYouNameYourself = readmatrix(..);

[Ludo Houben]

Hi Stephen23

Thanks for your code example. I'm going 'to play' with this solution.

Regarding to your comment:

"This statement I do not understand. When importing from e.g. CSV file you always use your variable names in your code.

ThisVariableYouNameYourself = readmatrix(..);"

'ThisVariableYouNameYourself' is not generating the needed overall structure. In the CSV file is NO variable naming and NO variable type information available. That's why I have to run the external created script.

Cheers

Ludo

Answer 1

[Voss]

% some variables:
a = rand(2,3,4);
x = 4;
simTimeData = zeros(226411,1);

% size to look for:
size_to_find = [226411,1];

% find it:
S = whos();
idx = find(cellfun(@(x)isequal(x,size_to_find),{S.size}))

idx = 3

S(idx)

ans = struct with fields:
          name: 'simTimeData'
          size: [226411 1]
         bytes: 1811288
         class: 'double'
        global: 0
        sparse: 0
       complex: 0
       nesting: [1×1 struct]
    persistent: 0

[Ludo Houben]

Hi Voss

Thanks for the answer. It looks like its working, but I don't understand the 'X' variable part. Is this a temporary variable? As you also have put it to the 'some variables'.

With kind regards

Ludo

[Stephen23]

"but I don't understand the 'X' variable part. Is this a temporary variable? As you also have put it to the 'some variables'."

There are two completely unrelated variables named x shown in that code:

• x in the base workspace, allocated to a scalar numeric. This is provided solely by Voss to demonstate that their code detects an array of the required size from amongst arrays of other sizes (e.g. the scalar x, the 3D a).
• x as the sole input argument to an anonymous function: https://www.mathworks.com/help/matlab/matlab_prog/anonymous-functions.html

They are unrelated, and for clarity should have different names.

Question out of curiosity: once you get the name of the variable with the right size, what do you do with that name?

[Ludo Houben]

Hi Stephen23

Thanks for the clarification. Just as I expected the 2nd 'X' was a temporary variable.

With that name I'm going to 'cherry pick' the desired data. The function that I'm creating is loading the previously created data (see answer to your question above about how I get the data) into the memory and will either find the 'named' variable as a variable in the memory, or as a part of a structure. In all cases it will use the data to create 'fixed' variables for start/step/end time for the simulation and export the data and time step information to the base workspace with fixed names.

This data is then connected to the main testing model. It the future there will be separate unit testing models and all unit testing models should be using the new variable names. Unfortunally I'm still cleaning up the current 'mess' of 6+ years 'developping' by non programmers. So this is a step to be able to support the current work methode of the engineers, but slowly transitioning them into a more structured way of working.

Cheers

Ludo

[Stephen23]

"With that name I'm going to 'cherry pick' the desired data"

How?

[Ludo Houben]

For *.mat files (Old crappy files that will be removed completely in the future)

simTimeData = MeasData.(variableList(idx).name);

The desired variable were created in the past and the workspace has a lot of unused variables. Or it can already contain the desired variables. This is the part where I'm currently working on.

For *.csv files (Export of the external tool)

simSignalData = eval(variableList(idx).name);   % I could not find a solution without the eval() yet

The desired variable was created by a earlier runned script (export from the external tool).

For *.m files (Files created in the new 'way of working' and/or manually created with data for unit tests)

run(iFile);     % iFile is the input of the function

The desired variables are already present and can be directly used.

For all files I will then create the following variables in the base workspace:

	% Create simulation variables in base workspace
		assignin('base','simSignalData', simSignalData);
		assignin('base','simTimeData', simTimeData);
		assignin('base','simCfgStepTime',simTimeData(2) - simTimeData(1));
		assignin('base','simCfgStartTime',simTimeData(1));
		assignin('base','simCfgStopTime',simTimeData(end));

In the furture I want to get rid of all the old crappy data 'things', but at this time its not possible to cleanup everything at once.

Cheers

Ludo

[Stephen23]

"For *.mat files ...The desired variable were created in the past and the workspace has a lot of unused variables. Or it can already contain the desired variables. This is the part where I'm currently working on."

LOAD into an output variable, then access its fields:

S = load(..);

"For *.csv files (Export of the external tool) simSignalData = eval(variableList(idx).name); % I could not find a solution without the eval() yet. The desired variable was created by a earlier runned script (export from the external tool)."

It is unclear why WHOS and EVAL would be required, you can import that CSV file data directly into a variable with a fixed name.

"For *.m files (Files created in the new 'way of working' and/or manually created with data for unit tests)... The desired variables are already present and can be directly used. For all files I will then create the following variables in the base workspace:"

Best avoided (see my tutorial). Much better: write a function and pass them as output arguments:

https://www.mathworks.com/help/matlab/matlab_prog/share-data-between-workspaces.html

[Ludo Houben]

"LOAD into an output variable, then access its fields:

S = load(..);"

I'm doing this, but I don't know the field name....that's the root cause of my challenge.

"It is unclear why WHOS and EVAL would be required, you can import that CSV file data directly into a variable with a fixed name."

The CSV file contains only the data, not the variable name and variable type information. So with only the CSV file I cannot create the required structure. At this time the external tool creates a script that holds this info, but I do not know the name of the variable that is used (See https://nl.mathworks.com/matlabcentral/answers/2041991-how-to-search-for-a-index-to-a-variable-with-known-size#comment_2947251 for more info)

"Best avoided (see my tutorial). Much better: write a function and pass them as output arguments:

https://www.mathworks.com/help/matlab/matlab_prog/share-data-between-workspaces.html"

This is what I'm trying to create :-)

Cheers

Ludo