MATLAB Answers

0

How can I programmatically control mouse motion and clicks with MATLAB?

I am creating a demo using MATLAB and I would like to programmatically move and click the mouse to demonstrate my application.

Products

5 Answers

Answer by MathWorks Support Team on 6 May 2015
 Accepted Answer

The ability to control mouse pointer position, motion and clicks is not available in MATLAB. To work around this issue, you can use the Java class java.awt.Robot which has this ability. Please note that this approach is not documented and hence MathWorks does not guarantee that the approach will be successful.
Below is a simple example that illustrates how you can move the mouse with MATLAB code using Java functionality. Note that we are unable to provide support for Java classes directly as they are not products of MathWorks. For further assistance regarding the Java classes and methods, refer to the following Java documentation page:
The following MATLAB code example demonstrates how one can programmatically control mouse motion using the java.awt.Robot class to move the mouse diagonally across the screen. First, import the class into MATLAB, create an object of this type, and then execute the mouseMove method in a loop to simulate motion.
 
import java.awt.Robot;
mouse = Robot;
mouse.mouseMove(0, 0);
screenSize = get(0, 'screensize');
for i = 1: screenSize(4)
mouse.mouseMove(i, i);
pause(0.00001);
end
The following example demonstrates how one can programmatically click the right mouse button to bring up the context menu. Again, import the required Java classes, create an object of this type, and then use the mousePress and mouseRelease functions to simulate a click. Before executing this code, place the mouse over a portion of the screen where a context menu can appear.
 
import java.awt.Robot;
import java.awt.event.*;
mouse = Robot;
mouse.mousePress(InputEvent.BUTTON3_MASK);
mouse.mouseRelease(InputEvent.BUTTON3_MASK);

  5 Comments

Sir can you please say how to perform drag operation we are doing final year project so that we want to implement drag operation you can contact me at this mail id:meda.revanth@gmail.com. My contact number is :7702227105 Send me reply so that it will be very helpful for us
I haven't tried this, but I believe calling mousePress, mouseMove, and mouseRelease should be the equivalent of click and drag. See the documentation linked in the MathWorks Support Team's answer for more information on those methods.
One interesting note - this works fine for me with a normal mouse configuration. If I change the mapping of mouse to pointer location (such as with Sakasa Mouse), the mouse.move command with java sends my mouse to a different location depending on where my mouse starts from, even if my command doesn't change. I found Chris Pyne's answer below got around this.

Sign in to comment.


Answer by Steve McClure on 12 Oct 2018
Edited by Steve McClure on 12 Oct 2018

As of R2018a, users can programmatically interact with their MATLAB Apps via the App Testing Framework: https://www.mathworks.com/help/matlab/app-testing-framework.html
This Developer Zone blog post also provides a tour of the functionality: https://blogs.mathworks.com/developer/2018/05/11/app-testing-framework/
The intended use for this framework is for testing (less so demonstrating) your apps, although it may suit your needs.

  0 Comments

Sign in to comment.


Answer by Amirhosein Ghenaati on 5 Dec 2014
Edited by Amirhosein Ghenaati on 5 Dec 2014

i used following code to simulate mouse left click action unfortunately the action command (press mouse) executes for ever and my status will be busy forever and i have to restart windows to work with matlab properly how can i fix this bug i want to simulate simple left click
.
import java.awt.Robot;
import java.awt.event.*;
mouse = Robot;
mouse.mouseMove(100, 100);
mouse.mousePress(InputEvent.BUTTON2_MASK); //left click press
mouse.mouseRelease(InputEvent.BUTTON2_MASK); //left click release

  4 Comments

Show 1 older comment
can anyone please help ?? as i am doing same project navigation control of powerpoint....actually i have set the zoom in position but the problem is it is not clicking the zoom in button..please help
Set a break point right after the mouseMove line and see if it positioned the mouse cursor in the correct position. Chances are it's not, so adjust your x,y values so that it does put it in the right position.
Very late reply, but for those reading this now: I copied Amirhosein's code and got wierd results (I wanted to select a program but ended up opening a new instance of the program instead). After a quick search I believe BUTTON2_MASK is the equivalent of clicking the middle (scroll) button. For left click you'd need:
mouse.mouseMove(810, 1060); % relates to my personal location of open program instance
mouse.mousePress(InputEvent.BUTTON1_MASK); % actual left click press
pause(0.1);
mouse.mouseRelease(InputEvent.BUTTON1_MASK); % actual left click release

Sign in to comment.


Answer by Chris Pyne on 10 May 2018

I am not sure about mouse clicks... but you can set the cursor position using the following code.
hroot=groot; % get the root handle
set(hroot,'PointerLocation',[400,600]); % default units are pixels so the domain is your screen resolution.

  1 Comment

This worked great! Thank you!

Sign in to comment.


Answer by Chuck
on 30 Apr 2016
Edited by Chuck
on 30 Apr 2016

Unfortunately, this is one area that MATLAB is behind other alternatives, such as Python. If you know how to code in Python, you can easily do it by using the following command:
import win32api, win32con
def click(x,y):
win32api.SetCursorPos((x,y))
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,x,y,0,0)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,x,y,0,0)
click(10,10)

  1 Comment

Hi!
I searched the method for the mouse left click detection and I find this great way to solve my issue using Windows API.
I need to get the x and y coordinates of mouse left click of the area, that isn't a figure or current MATLAB window. By this time, I can get x and y of pointer using groot. But I still didn't find an apropriate way to detect the mouse left click. I think that WM_LBUTTONDOWN message of Windows API can do it. But by this moment unfortunatly I didn't find how to program the WinAPI commands into MATLAB code.
Can you explain me how to do it well?

Sign in to comment.