we are facing problem with assigning the functions of mouse to detected color and using java.awt.robot. the code is as follows ...so plz help

% Capture the video frames using the videoinput function % You have to replace the resolution & your installed adaptor name. vid = videoinput('winvideo',1,'YUY2_352x288');

% Set the properties of the video object set(vid, 'FramesPerTrigger', Inf); set(vid, 'ReturnedColorspace', 'rgb') vid.FrameGrabInterval =7;

%start the video aquisition here start(vid)

% Set a loop that stop after 100 frames of aquisition while(vid.FramesAcquired<=500)

    % Get the snapshot of the current frame
    data = getdata(vid,1);
    datai = flipdim(data,2);
    % Now to track red objects in real time
    % we have to subtract the red component 
    % from the grayscale image to extract the red components in the image.
    diff_im_red = imsubtract(datai(:,:,1), rgb2gray(datai));
    diff_im_blue = imsubtract(datai(:,:,3), rgb2gray(datai));
    diff_im_green = imsubtract(datai(:,:,2), rgb2gray(datai));

    %Use a median filter to filter out noise
    diff_im_red = medfilt2(diff_im_red, [3 3]);
    diff_im_blue = medfilt2(diff_im_blue, [3 3]);
    diff_im_green = medfilt2(diff_im_green, [3 3]);

    % Convert the resulting grayscale image into a binary image.
    diff_im_red = im2bw(diff_im_red,0.18);
    diff_im_blue = im2bw(diff_im_blue,0.18);
    diff_im_green = im2bw(diff_im_green,0.18);

    % Remove all those pixels less than 300px
    diff_im_red = bwareaopen(diff_im_red,300);
    diff_im_blue = bwareaopen(diff_im_blue,300);
    diff_im_green= bwareaopen(diff_im_green,300);
    % Label all the connected components in the image.
    bw_red = bwlabel(diff_im_red, 8);
    bw_blue = bwlabel(diff_im_blue, 8);
    bw_green = bwlabel(diff_im_green, 8);

    % Here we do the image blob analysis.
    % We get a set of properties for each labeled region.
    stats_red = regionprops(bw_red, 'BoundingBox', 'Centroid');
    diff_im_red = imsubtract(data(:,:,1), rgb2gray(data));
    stats_blue = regionprops(bw_blue, 'BoundingBox', 'Centroid');
    diff_im_blue = imsubtract(data(:,:,3), rgb2gray(data));
    stats_green = regionprops(bw_green, 'BoundingBox', 'Centroid');
    diff_im_green = imsubtract(data(:,:,2), rgb2gray(data));

    % Display the image
    imshow(datai)

    hold on

    %This is a loop to bound the red objects in a rectangular box.
    for object_red = 1:length(stats_red)
        bb_red = stats_red(object_red).BoundingBox;
        bc_red = stats_red(object_red).Centroid;
        rectangle('Position',bb_red,'EdgeColor','r','LineWidth',2)
        plot(bc_red(1),bc_red(2), '-m+')
        a_red=text(bc_red(1)+15,bc_red(2), strcat('X: ', num2str(round(bc_red(1))), '    R: ', num2str(round(bc_red(2)))));
        set(a_red, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'RED');       
end

     for object_blue = 1:length(stats_blue)
        bb_blue = stats_blue(object_blue).BoundingBox;
        bc_blue = stats_blue(object_blue).Centroid;
        rectangle('Position',bb_blue,'EdgeColor','b','LineWidth',2)
        plot(bc_blue(1),bc_blue(2), '-m+')
        a_blue=text(bc_blue(1)+15,bc_blue(2), strcat('X: ', num2str(round(bc_blue(1))), '    B: ', num2str(round(bc_blue(2)))));
        set(a_blue, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'BLUE');
     end

    for object_green = 1:length(stats_green)
        bb_green = stats_green(object_green).BoundingBox;
        bc_green = stats_green(object_green).Centroid;
        rectangle('Position',bb_green,'EdgeColor','G','LineWidth',2)
        plot(bc_green(1),bc_green(2), '-m+')
        a_green=text(bc_green(1)+15,bc_green(2), strcat('X: ', num2str(round(bc_green(1))), '    G: ', num2str(round(bc_green(2)))));
        set(a_green, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'green');    
    end
    hold off    
end

% Both the loops end here.

% Stop the video aquisition. stop(vid);

%

Answer 1

[Image Analyst]

Decide what you want to do, then call the move or click events with the right parameters.

import java.awt.Robot
import java.awt.event.*
mouse = Robot;

% Press a push button on the page.
% Move to pixel (800, 405) on the computer screen.
mouse.mouseMove(800, 405); % First message.
% Click the left mouse button once.
mouse.mousePress(InputEvent.BUTTON1_MASK);
mouse.mouseRelease(InputEvent.BUTTON1_MASK);

See http://undocumentedmatlab.com/blog/gui-automation-robot

http://docs.oracle.com/javase/6/docs/api/java/awt/event/MouseEvent.html

http://docs.oracle.com/javase/6/docs/api/java/awt/event/InputEvent.html

[Ajay Gawli]

Now we need to assign the scrolling function of mouse to the detected green object .Could u helpus with it .

Answer 2

[Image Analyst]

I'm not sure why or where you want to use java robot. To have the color of the pixel under your mouse curcor displayed in real time in a text panel on your GUI, use the impixelinfo() function.

[Ajay Gawli]

We have detected the red blue and green objects using the code which is mentioned above .now to those detected red green blue objects ..we wish to assign the functions of mouse ...like if red object is detected then the cursor should move as the object moves ....green object to scroll...and one blue object means left click ...two blue objects means right click and ...three blue objects means double click ....so could u make sone changes in the above code and show us how to do it or plz guide us through

[Image Analyst]

I can't write more for you. If you want someone to write the program for you, you can hire someone. I did guide you though some of it, like moving the mouse cursor to a specific place on the screen and clicking the mouse button. And I did guide you to other links that guide you into how you can simulate other mouse and keyboard functions. I don't know how to simulate rolling the mouse wheel to scroll the display. It's probably in the directions somewhere but I would have to look it up, but you can do that just as well as I can or even better.

I can give you a demo where I track movement of a green Sharpie marker in a video, and that is attached.

I think I've guided you to all the tools you'll need to complete your project. There is nothing particularly difficult after this. It's just a simple matter of putting together all the plumbing in your code to use it (for loops if statements, etc.), so you should be able to do that on your own. Good luck on completing your project with these tools and demos. If my answer helped, then perhaps you could "Accept" or "Vote" for it.