I can figure out two solutions
1) check if the simulation program has API for running it from "outside". In this case, make reference to the program API documentation
2) if you can run the symulation program from command line and it accepts files as input/output: in your matlab objective function, write a text file with the degrees of freedome values selected by the Matlab optimization algorithm you are using, then run the simulation program and let it read the input file, run the simulaiton, write another text file with the output and read the output from the matlab objective function.
function f = myObjFunction(x)
fid = fopen('input.txt');
command = '...'
status = system(command);
fid = fopen('output.txt');
Reading and writing text files can take a lot, but if the complexity is dominated by the simulation, this method could worth a try