Handle to an external window from MATLAB

Hello there,

I am opening an external exe file from MATLAB using the command !program.exe &

Issue 1: Everytime I open this program (in Windows mode 1280*720 and my screen size is 1366*768), it opens at different locations of the screen which makes it difficult for me to do certain processing automatically.

Requirement: Is there a method to solve issue 1

Actually, I want to access the screen size, contents and play around with the new external window (program.exe). I believe getting a handle to this could solve my problem. Any thoughts?

Answer 1

[Walter Roberson]

http://stackoverflow.com/questions/1888863/how-to-get-main-window-handle-from-process-id

[Madhu Kodappully]

Thanks. I appreciate your posting. What would be the MATLAB equivalent for EnumWindows() and GetWindowThreadProcessID().

[Walter Roberson]

http://www.mathworks.com/matlabcentral/newsreader/view_thread/9440

http://www.mathworks.com/matlabcentral/answers/44236-how-to-determine-the-location-of-the-window-of-other-application-not-matlab-and-its-size

http://www.mathworks.com/matlabcentral/fileexchange/29594-textinject

[Madhu Kodappully]

Thanks again.

I created a MATLAB GUI from which I called program.exe and it opens in an external window. Working all nice and great. I am still unable to get the contents on the external window screen (I badly need those contents). Is there a code that helps to read contents from an external window into a text file?

What I would really like to have is an actxcontrol that can open program.exe within the GUI. I do not know whether such an actxcontrol is available (for example, I can open internet explorer from a MATLAB gui using appropriate actxcontrol).

[Madhu Kodappully]

I have obtained the process id of the application window program.exe.

[response,tasks] = system('tasklist/fi "imagename eq program.exe"');
pid_process = str2num(char({tasks(186:189)}));

Now, how do I use this process ID to read the contents inside this application window? It has several text and numeric data which I want to store into a text file.

Please share your thoughts

[Walter Roberson]

My thought is "Don't use MS Windows". I am googling this stuff as I go along.

MATLAB is not designed as an operating-system level interface. There are lot of things you cannot do directly in MATLAB and need to call into C/C++ or use an ActiveX control for.

You are trying to read the content of a window in a process that does not belong to you. That is a a security violation if done without the cooperation of the other program. A quick glance around suggests that you need to use the MS Windows specific WM_GETTEXT. There appears to be an example at http://stackoverflow.com/questions/7740379/c-sharp-how-to-use-wm-gettext-getwindowtext-api

[Madhu Kodappully]

I appreciate your posting. MATLAB has actxcontrol for Internet Explorer and I have found its content reading possible. I am looking for a similar actxcontrol for running exe files within MATLAB GUI.

I have seen people attempting dynamic data exchange between matlab and excel application. Any thoughts along that line?

[Walter Roberson]

activex controls including dynamic data exchange depend upon the recipient program having been programmed to expect such things and act upon them.

The Unix way of handling this kind of thing would be for the parent process to open a pair of pipes before forking the process, which would be inherited as file descriptors by the child process. Then they each read from one of the pipes and writes to the other.

The alternative Unix way of handling this kind of thing would be through TCP/IP: a program that wanted to be able to receive queries from other programs would establish itself as a TCP server, and any process that wanted to talk to it would be a TCP client.

[Madhu Kodappully]

Thanks for this information. The method illustrated below turned out to be a productive solution:

1. I saved a reference image from the screen as .jpg file.

2. Now, as the program.exe runs, I used a timer in MATLAB to capture the image appearing in the same area on the screen from where the reference image is previously captured please note: The program always run in maximized window and there is no conflict of location [screen_x screen_y image_x image_y].

3. I compared the newly obtained image with the already saved reference image.

4. Tried histogram method, PSNR and mse method, SSIM method

5. When I do a single comparison of two images, things work fine

6. When I am doing the timer based continuous comparison, I am not getting a the end result match even though I am seeing the screen showing the exact image I am looking for.

Any thoughts?

[Walter Roberson]

It is a security violation to allow one program to take an image snapshot of another without the cooperation of the other program. Therefore when MATLAB asks for the screen contents, the operating system can (and should) return either nonsense or only the contents of the MATLAB windows and the desktop icons.

Security too often only appears to be a priority with Microsoft when it comes to preventing people from playing DVDs, so you might be able to do better in practice, but you would be better spending your time on arranging for the programs to cooperate.

Question: is the external program a web browser? If it is then there are tools available from MATLAB to run a browser within a context that can be examined.

[Madhu Kodappully]

Hi there,

Thanks for your response. As I have already updated in my previous responses, it is a *.exe application which I am running from MATLAB. Unfortunately I did not find any suitable actxcontrol method to run the exe within the MATLAB GUI.

In the image acquisition method, I have used subplot(2,1,1) and subplot(2,1,2) to check the authenticity of the images. Both the images are from the exe window only and not any other image as you suspect.

[Walter Roberson]

If you save the images with the timer and then run them through the comparison one by one, then do you get a match? If Yes but not when you are running the comparison within the timer callback then that suggests that you have a problem in the callback.

[Madhu Kodappully]

Dear Walter,

Thanks for your reply. 1. I have my reference image already saved.

2. The new image is a crop from screencapture

3. Then I performed image comparison

4. Checked for peak signal to noise ratio

5. That is the end result

Everything works nice ....