Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Virtual Mouse Project

Asked by Syahrul Fikri on 16 Mar 2012

Hey everyone, I'm doing a virtual mouse project by controlling the mouse pointer on the computer screen using a webcam to track movement of my hand n some mouse clicking event. I have to learn matlab from scratch n I've tried doing as possible as I can but still can't produce the right codes or thru simulink. Can anyone really help me on this bcuz I'm kinda desperate right now n I'm really don't know what I'm suppose to do next. If anyone can give me the codes or simulink models, that would be really helpful, thank u..

7 Comments

Syahrul Fikri on 6 Apr 2012

oh, that will truly grateful, thanx man, i used the simulink correlation demo as a guide, which is available in the matlab demos, here is the link to the picture, http://www.flickr.com/photos/43454132@N04/7050008053/in/photostream thanx again, anyone who can give any ideas are very much appreciated

Syahrul Fikri on 12 Apr 2012

final step people!! how do i convert my virtual mouse m.file to exe.file?? i tried using the deployment project method - file/new/deployment project.. , all i got is error - >> coder('-build', 'Virtual Mouse (Red Object Tracking).prj') ??? This text contains non-empty top-level expressions. It appears to be a script. Error in ==> tracking_red Line: 1 Column: 1 Code generation failed: Open error report. - my codes are below for checking

Walter Roberson on 12 Apr 2012

You can only compile function .m, not script .m . Add a "function" line to your .m file.

Syahrul Fikri

Products

No products are associated with this question.

8 Answers

Answer by Matt Kindig on 4 Apr 2012

Controlling the mouse pointer is actually a bit difficult, as you need to use the Java Robot class (it's not a Matlab function per se). I'm not even sure how you would access this via Simulink.

See below: http://www.mathworks.com/support/solutions/en/data/1-2X10AT/index.html?solution=1-2X10AT

1 Comment

Walter Roberson on 4 Apr 2012

Example Java Robot use of keyboard: http://www.mathworks.com/matlabcentral/answers/5259-how-to-simulate-keyboard-key-inputs

The Solution that Matt shows above has an example of mouse control.

You could use a MATLAB Function block to access this kind of code; you would create the robot for the initialization phase and give commands to the robot based upon the run-time input signals.

Matt Kindig
Answer by Teja Muppirala on 4 Apr 2012

MATLAB can move the mouse around:

x = 500;
y = 600;
set(0,'PointerLocation',[x y]);

But, I think you do need the Java Robot to make a click.

1 Comment

Walter Roberson on 4 Apr 2012

Unless you can figure out which object you are over and locate its callback function and activate it, passing in a suitable "event" structure to represent right-click, shift-click etc.

Teja Muppirala
Answer by Syahrul Fikri on 7 Apr 2012

hye all, ive found this one m file code from the matlab central file exchange, this code is to track red object in real time image acquisition, my question is, how do u connect the tracking bounding box to the mouse input from the java robot library?? the following are the codes

% Capture the video frames using the videoinput function % You have to replace the resolution & your installed adaptor name.

vid = videoinput('winvideo',1,'RGB24_320x240');

% Set the properties of the video object

set(vid, 'FramesPerTrigger', Inf); set(vid, 'ReturnedColorspace', 'rgb') vid.FrameGrabInterval = 5;

%start the video aquisition here

start(vid)

% Set a loop that stop after 100 frames of aquisition

while(vid.FramesAcquired<=200)

    % Get the snapshot of the current frame
    data = getsnapshot(vid);
    % Now to track red objects in real time
    % we have to subtract the red component 
    % from the grayscale image to extract the red components in the image.
    diff_im = imsubtract(data(:,:,1), rgb2gray(data));
    %Use a median filter to filter out noise
    diff_im = medfilt2(diff_im, [3 3]);
    % Convert the resulting grayscale image into a binary image.
    diff_im = im2bw(diff_im,0.18);
    % Remove all those pixels less than 300px
    diff_im = bwareaopen(diff_im,300);
    % Label all the connected components in the image.
    bw = bwlabel(diff_im, 8);
    % Here we do the image blob analysis.
    % We get a set of properties for each labeled region.
    stats = regionprops(bw, 'BoundingBox', 'Centroid');
    % Display the image
    imshow(data)
    hold on
    %This is a loop to bound the red objects in a rectangular box.
    for object = 1:length(stats)
        bb = stats(object).BoundingBox;
        bc = stats(object).Centroid;
        rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
        plot(bc(1),bc(2), '-m+')
        a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), '    Y: ', num2str(round(bc(2)))));
        set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
    end
    hold off
end

2 Comments

Walter Roberson on 7 Apr 2012

x = bc(1);
y = bc(2);

and then use the technical solution Matt referred to.

Syahrul Fikri on 7 Apr 2012

the code works!!! thank u so much for all your help walter, im truly grateful, thanx a zillion times

Syahrul Fikri
Answer by vaibhav mathur on 7 Apr 2012

put the value of red object's centroid in the [x y] mat. and set(0,'PointerLocation',[x y]); and the pointer moves according to centroid may be it work good luck!

0 Comments

vaibhav mathur
Answer by Syahrul Fikri on 7 Apr 2012

hye all, i would like to say thank u so much for all your help n suggestions, finally, my code works by using the red tracking code n connect it to the java robot, thanx again all, if i encounter further problems, ill post it here, :D

0 Comments

Syahrul Fikri
Answer by Syahrul Fikri on 8 Apr 2012

Hye all, from the code above, I've successfully execute it with the java robot command, I hv a question, can anyone explain to me how can a mouse clicking event being execute when the program detects 2 tracking red object, the code can detect 2 red object, my plan is the first red object is to move the mouse pointer, but when it detects the second red object, a mouse clicking event is execure

3 Comments

Walter Roberson on 8 Apr 2012

mousePress

http://docs.oracle.com/javase/1.4.2/docs/api/java/awt/Robot.html#mousePress%28int%29

Syahrul Fikri on 9 Apr 2012

ok.., i know about the mouse press n release java code, ive implemented it, but the mouse press n release is always on whenever i move my pointer, do u know how to write the command of the mouse press n release code being execute only when my code track the second red object?? im still learning the red tracking codes by the way, :D, here is my code so far:

% Capture the video frames using the videoinput function
% You have to replace the resolution & your installed adaptor name.
vid = videoinput('winvideo',1,'RGB24_640x480');

% Set the properties of the video object
set(vid, 'FramesPerTrigger', Inf);
set(vid, 'ReturnedColorspace', 'rgb')
vid.FrameGrabInterval = 5;

%start the video aquisition here
start(vid)

% Set a loop that stop after 100 frames of aquisition
while(vid.FramesAcquired<=20000)

% Get the snapshot of the current frame
data = getsnapshot(vid);

% Now to track red objects in real time
% we have to subtract the red component
% from the grayscale image to extract the red components in the image.
diff_im = imsubtract(data(:,:,1), rgb2gray(data));
%Use a median filter to filter out noise
diff_im = medfilt2(diff_im, [3 3]);
% Convert the resulting grayscale image into a binary image.
diff_im = im2bw(diff_im,0.18);

% Remove all those pixels less than 300px
diff_im = bwareaopen(diff_im,300);

% Label all the connected components in the image.
bw = bwlabel(diff_im, 8);

% Here we do the image blob analysis.
% We get a set of properties for each labeled region.
stats = regionprops(bw, 'BoundingBox', 'Centroid');

% Display the image
imshow(data)

hold on

%This is a loop to bound the red objects in a rectangular box.
for object = 1:length(stats)
bb = stats(object).BoundingBox;
bc = stats(object).Centroid;
rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
plot(bc(1),bc(2), '-m+')
a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), ' Y: ', num2str(round(bc(2)))));
set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
import java.awt.Robot;
import java.awt.event.*;
mouse = Robot;
mouse.mouseMove(bc(1), bc(2));
mouse.mousePress(InputEvent.BUTTON2_MASK);
mouse.mouseRelease(InputEvent.BUTTON2_MASK);
end

hold off
end

vidit virmani on 19 Feb 2013

hey.... i feel lucky to find this page and the above discussion.... because i am too working on the same project of controlling mouse by color detectin methods.... i used the same code to detect a color.. i used blue instead of red.. i planned to execute the clicking events by manipulating the distance calculated between my finger and the thumb(both covered with a blue tape).. how can i do that....?? . . %% import java.awt.Robot; import java.awt.event.*; mouse = Robot; mouse.mouseMove(bc(1), bc(2)); mouse.mousePress(InputEvent.BUTTON2_MASK); mouse.mouseRelease(InputEvent.BUTTON2_MASK); %% do i have to use this code directly with the color detection code(as u have stated in above comment) or some prerequisites are to b done to interface java with MATLAB..?? please guide me for i am a new matlab user...

Syahrul Fikri
Answer by vidit virmani on 21 Feb 2013

@syahrul fikri : i used your code and its working... the mouse pointer moves and also clicks, but the problems i faced are..... 1) i dont know how the clicking operation is being executed.... does it executes when pointer is stable on a folder for some specified time or when another color is detected by the system?...... 2) if i move my hand to right , the pointer moves left and vice versa..... how to reverse that....... 3)if the camera resolution is less than the resolution of the screen, whole screen is not covered with this system.... how to eliminate this?..... 4)if i want to detect two different colors and execute clicking operations by manipulating the distance between the colors, how can i do that..?? please help....

1 Comment

Rajshekar on 24 Apr 2013

@vidit: 2) this is normal mirror concept: hence the pointer position shall have the formula of [1024-x, 780-Y] where x and y are the original positions in your code. the numbers 1024 AND 780 SORT OF RESOLUTION OF YOUR SCREEN.It depends on your screen resolution.

vidit virmani
Answer by Rajshekar on 24 Apr 2013

Hey i tried to check this interesting thing: i used java robot.was able to move pointer and click as well. But there is this thing : when i right clicked a block in simulink(ofcourse by java robot), menu drop down appeared as normal; in the next lines of codes, i meant to position cursor on one of the menu item so i tried to reposition and use right click to select the menu item. But by the time these lines were to execute, the menu was not visible and hence the menu item cannot be selected

i think the menu items disappears when next lines of codes starts execution because the script window activates.

Do you know how to avoid this disappearance of the menu.

0 Comments

Rajshekar

Contact us