How can I change the resolution of kinect V2 in matlab?

Kinect V2 color stream supported format is : 1920x1080. But kinect V2 depth stream format is : 512x424. Now when I start live steam for both sensors then they have different sizes because of different resolution. I cant resize them, because I need coordinates . so when I resize using Imresize(),the coordinates are not matched. I already read matlab documentation.They said hardware only supports this two format respectively.Now How can i do this in code so that both stream have the same resolution. I tried two days long but failed.

Answer 1

[Walter Roberson]

According to http://smeenk.com/kinect-field-of-view-comparison/ the fields of view of the color sensor and depth sensor are different, 84.1 x 53.8 degrees compared to 70.6 x 60 degrees, so having the depth sensor return the same resolution as the color sensor (or an integer reduction thereof) would be misleading.

The solutions appear to be discussed https://www.mathworks.com/matlabcentral/answers/268152-mapping-rgb-and-depth-kinect-v2

[sufian ahmed]

walter bro can u tell me the step what i need to do now. i just stuck on this problem. I just want what i need to do if i want to set same resolution for both depth and color sensor of kinect.

[Walter Roberson]

It is not possible to change the resolution of the depth or color sensor. You can crop and resize what you get after you get it.

The discussion I linked to above talks about doing the necessary coordinate conversion.

[sufian ahmed]

coordinate conversion how can i do..? I actually dnt understand their coversation.

[Walter Roberson]

They use the camera calibration to undistort the image. Then they take the coordinates of each point in the depth image and use a transformation matrix to project it into the coordinate system of the color image. Then they run through each point and the ones that would end up inside a rectangle with the same coordinates as the color image, they copy over from the original depth image into the new coordinates.

This technique is known as a "push" mapping, and push mappings are inferior to "pull" mappings (take coordinates of the locations on the color image, calculate where they are in depth image coordinates, interpolate from the nearby points) but it can be useful.

None of this can happen without camera calibration.

[sufian ahmed]

bro without ckboard can i calibrate ? can u provide me the complete code please. i tried their code but it not works :(

[Walter Roberson]

"without ckboard can i calibrate"

You could use the facilities of the stereo camera calibrator app, but you will run into difficulties because the kinect visual and depth camera pair is not really a stereo camera setup. https://www.mathworks.com/help/vision/ug/stereo-camera-calibrator-app.html

I see some tools for doing kinect calibration together with MATLAB; see

http://arthronav.isr.uc.pt/~carolina/kinectcalib/

https://sourceforge.net/projects/kinectcalib/

https://github.com/xw1120/Joint-Calibration-Toolbox-for-Kinect

These vary in how many images are needed (6 or 30), and only one of the above supports offline calibration. But all three appear to use checkerboards for calibration.

It is theoretically possible to calibrate without a checkerboard, if you put several objects into the field of view that are each of known size and known position. You need to use several objects per view because there is non-uniform sensor distortion that needs to be calibrated across the field of view.

[sufian ahmed]

this calibration tool for kinect v1 but i have v2. walter bro please clear this. why i need to do calibration. By doing calibration what value i need to take? and then how can i use those values to get same resolution image from both sensor. please clear this and please explain the process step by step so that i can get same resolution image including same coordinates. i am fully drowning in confusion

[Walter Roberson]

My only PC has been sitting in my junk pile for the last decade, so I cannot experiment with the Kinect v2 that I do not have -- but I would imagine that it is not much work to change the v1 source code to support v2.

"why i need to do calibration."

I do not care whether you calibrate or not. It is your project, not mine.

Imagine you have two cameras looking in vaguely the same direction, but one of the two of them has its lens higher and off to the side of the other, and one of the two of them has a fish-eye lens, and one of the two of them is higher resolution than the other. And you want to bring them into the same resolution for what-ever reason.

Now you could bring them into the same resolution by just doing an imresize() on one of the two of them to match the size of the other. Would that be useful? Well, it would allow you to proceed with your computations, which I suppose might be useful. But any one point in one of the two images would be only "sort of near" the point with the same row and column in the other image, if "sort of near" means something like "in the same quarter of the room". Distances in images taken with fish-eye lens are non-linear, with the physical points corresponding to adjacent pixel centers getting further and further apart as you get further from the centre of the image. You might say "But the centre of the image is Okay, right?" but remember that the centre of one of the images is not the same physical point as the centre of the other image, because the lens are not pointed at the same place.

You could also bring the two cameras into the same resolution by just doing an imcrop() to match the sizes. You will find that will would be about as useful as doing an imresize(), except with the additional tension about where you crop.

Calibration is the process of figuring out which points in the image of one camera correspond to which points in the image of the other camera so that the images can be transformed to be brought into alignment, just in case you wanted to do something potentially useful like matching the color of a physical location to the depth at that same physical location.

"please explain the process step by step"

The step by step process is to get the source code for the first toolkit I linked to, and make the minor code modifications to use Kinect v2, and then follow the steps in the manual.

"walter bro"

Note: I really do not appreciate being referred to as "bro". I am almost certainly older than your mother.

[sufian ahmed]

oh. sorry for that, walter uncle.

[Jonathan]

Hello @Walter Roberson

I am using Kin2 Toolbox for Matlab for the Kinect V2 sensor which I got from file exchange https://www.mathworks.com/matlabcentral/fileexchange/53439-kinect-2-interface-for-matlab. There is a code for mapping the color to depth that I am trying to use. I want to determine an red object's centroid and get the depth of that object at that pixel so that I get the distance from the sensor to the object. These will then be sent to a robot for a pick and place operation. Below is the code. However the code takes only one frame and keeps on repeating those centroids' coordinates after every five seconds and doesn't update. What could I be doing wrong? Other frame acquiring functions like getsnapshot, snapshot or step are giving errors saying because its uint8 or struct. Could you please help me figure that out?

addpath('Mex');
clear all
close all
redThresh = 0.25;

k2 = Kin2('color','depth');
% images sizes
d_width = 512; d_height = 424; outOfRange = 4000;
c_width = 1920; c_height = 1080;

% Color image is to big, let's scale it down
COL_SCALE = 0.5;

% Create matrices for the images
depth = zeros(d_height,d_width,'uint16');
color = zeros(c_height*COL_SCALE,c_width*COL_SCALE,3,'uint8');

% Images used to draw the markers
depthAdditions = zeros(d_height,d_width,3,'uint8');
colorAdditions = zeros(c_height*COL_SCALE, c_width*COL_SCALE,3,'uint8');

% depth stream figure
d.h = figure;
d.ax = axes('units','pixels');
d.im = imshow(depth,[0 255]);
title('Depth Source')
% color stream figure
c.h = figure;
c.im = imshow(color,[]);
title('Color Source');

nFrame = 0;
while true
    % Get frames from Kinect and save them on underlying buffer
    validData = k2.updateData;
    
    % Before processing the data, we need to make sure that a valid
    % frame was acquired.
    if validData
        % Copy data to Matlab matrices
        depth = k2.getDepth;
        color = k2.getColor;
        
        % update depth figure
        depth8u = uint8(depth*(255/outOfRange));
        depth8uc3 = repmat(depth8u,[1 1 3]);
        set(d.im,'CData',depth8uc3 + depthAdditions);
        
        % update color figure
        color = imresize(color,COL_SCALE);
        set(c.im,'CData',color + colorAdditions);
        
    end
    
    while (nFrame<300)
      
        frame=color();
        red = frame(:,:,1);
        diffFrame = imsubtract(frame(:,:,1), rgb2gray(frame));
        diffFrame = medfilt2(diffFrame, [3 3]);
        binFrame = imbinarize(diffFrame, redThresh);
        binFrame = bwareaopen(binFrame,300);
        bw = bwlabel(binFrame, 8);
        
        stats = regionprops((bw), 'BoundingBox', 'Centroid');
        centroids = cat(1, stats.Centroid);
        imshow(frame)
        %drawnow
        hold on
        
        %This is a loop to bound the red objects in a rectangular box.
        for object = 1:length(stats)
            bb = stats(object).BoundingBox;
            bc = stats(object).Centroid;
            rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
            plot(bc(1),bc(2), '-m+')
            a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), '    Y: ', num2str(round(bc(2)))));
            set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
            X = bc(1); Y = bc(2);
           
            x = X;
            y = Y; 
            disp('Input Color Coordinates')
            disp([x y]);
            
            depthCoords = k2.mapColorPoints2Depth([x/COL_SCALE y/COL_SCALE]);
            
            disp('Output depth coordinates')
            disp(depthCoords);
            depthAdditions = insertMarker(depthAdditions,depthCoords,'Color','green');
            
        end
        hold off
        pause(5)
        
        nFrame = nFrame+1;
    end
end

k2.delete
close all

[Walter Roberson]

Please show those error message you mention, along with the commands that are generating the errors.

[Jonathan]

The following is an error generated when I use getsnapshot or snapshot or getdata

Undefined function 'getsnapshot' for input arguments of type 'uint8'.

Error in RgbColorMappingTrial (line 82)

s=getsnapshot(color);

This comes when I use :

     while (nFrame<300)            
            s=getsnapshot(color);% when I use getsnapshot/getdata/snapshot on this line.
            red = s(:,:,1);
            diffFrame = imsubtract(s(:,:,1), rgb2gray(s));
            diffFrame = medfilt2(diffFrame, [3 3]);
            binFrame = imbinarize(diffFrame, redThresh);
            binFrame = bwareaopen(binFrame,300);
            bw = bwlabel(binFrame, 8);

The code I gave earlier is determining the centroids and mapping to depth giving depth coordinates but it continuously give for the same frame over and over again. I do not know why.

[Walter Roberson]

You need to fetch the color data from k2 not from your array named "color"

[Jonathan]

Thank you sir. I however do not seem to understand how to do what you mentioned above. I have checked the Kin2 code but I dont really seem to understand how to go about it. Would you show in terms of code how I can be able to go about it.

[Walter Roberson]

maxFrame = 300;
vid = videoinput('kinect', 1);   %1 for color, 2 for depth
vid.FramesPerTrigger = 1;
vid.TriggerRepeat = maxFrame;
start(vid);

for nFrame = 1 : maxFrame
    s = getdata(vid);
    red = s(:,:,1);
    [...]
end

[Jonathan]

Thank you Mr Roberson. It worked. However, I want to ask if it is possible that for every RGB frame taken, can I be able to take a depth image at the same time? I want that for every 5 seconds that the rgb camera is taking an image, the depth camera must also take the image of the same frame at the same time. Can that be possible? If so can you please show me how I can achieve that.

Thank you.

[Walter Roberson]

maxFrame = 300;
vid = videoinput('kinect', 1);   %1 for color, 2 for depth
vid.FramesPerTrigger = 1;
vid.TriggerRepeat = maxFrame;
depth = videoinput('kinect', 2);   %1 for color, 2 for depth
depth.FramesPerTrigger = 1;
depth.TriggerRepeat = maxFrame;
start(vid);
start(depth)
for nFrame = 1 : maxFrame
    s = getdata(vid);
    d = getdata(depth);
    red = s(:,:,1);
    [...]
end