Simulink Support for Kinect

Simulink Support for Kinect is a set of Simulink blocks interfaced with OpenNI and Kinect SDK.

Description

Simulink Support for Kinect consists of IMAQ, Image, IR, Depth, Motion and Skeleton blocks to achieve real-time image data acquisition from Kinect devices such as Microsoft Kinect or ASUS Xtion PRO under Simulink simulation environment.
It works with 32bit/64bit version of MATLAB R2010b or later in Windows7 32bit/64bit and it requires the following 3rd party software:

[C MEX Compiler]
- Microsoft Visual Studio 2010 Express Edition (VC++)
- Microsoft Windows SDK 7.1

[To use OpenNI (http://www.openni.org/)]
- OpenNI
- PrimeSense NITE
- PrimeSensor Module

[To use Microsoft Kinect SDK (http://www.microsoft.com/en-us/kinectforwindows/)]
- Microsoft Kinect for Windows SDK v1.6

For more detailed information, please refer to document in the zip file and help of the blocks.

[Optional MATLAB products]
- Computer Vision System Toolbox (highly recommended)
- DSP System Toolbox (to acquire Kinect Microphone Array data)
- Simulink Coder (to generate *.exe file)

Acknowledgements

Kinect Matlab and Kinect Microsoft Sdk inspired this file.

This file inspired Simulink For Pcv (Point Cloud Viewer).

• Required Products: Simulink
• MATLAB release: MATLAB 7.11 (R2010b)

[Christian]

Hello everybody,

First of all thanks Takashi for this great submission.
Second, I successfully ran SETUP_KINECTSDK and setup_openni.m, but when I try to run some example this message appears.

Error reported by S-function 'sfun_nid' in 'nid_cvst_image/NID IMAQ':

NID Error: Failed to initialize NID. It might be caused by
- NID is not connected.
- NID unsupported block is used in the model
- NID unsupported IMAQ Resolution and FPS is configured.

Some clue to solve this.

Thanks in advance

Christian

[Samuel]

Dan,

You could connect the signal output of the Skeleton block to a "To workspace" Simulink block. This will save the output of your simulation to your current MATLAB workspace. You can then access joint data as a 20x3xN matrix (where N is the number of frames in your simulation and assuming you are using the Windows kinect runtime).

[Dan]

Thank you for providing this code. Great resource. Question, how would you go about saving the XYZ for each joint? I am not familiar with Simulink. Thanks

[Takashi Chikamasa]

Hi all,

Please make sure about the following steps to solve installation issues:
1. Make sure that all SDK (VC++, Windows SDK, Kinect SDK/OpenNI) are installed properly
2. Make sure that VC++ is set as mex compiler
>> mex -setup
3. Read document in the Simulink support Kinect.

Additionally, I recommend to install MATLAB under the directory without spaces because of mex compilation with third party SDKs.

Concerning Windows8, I have never used Win8, so unfortunately, I can't answer your question.

Takashi

[Maleeha]

Hi Takashi,
I am trying to run the setup_kinectsdk.m file on the Matlab, however it gives me the following error
"
>> setup_kinectsdk
### Confirming Simulink Support for Kinect installations with Kinect SDK...
### Setting MATLAB path to .\Lib and .\Lib\doc_en directories.
### Generating C:\Users\Shahzada\Documents\MATLAB\simulinkfornidinfo.m
### Generating C MEX file for Simulink Support for Kinect...

C:\PROGRA~1\MATLAB\R2012A\BIN\MEX.PL: Error: 'sfun_nid.cpp' not found.

Error using mex (line 206)
Unable to complete successfully.

Error in makecmexfile4nid (line 62)
mex('sfun_nid.cpp', 'kinectSDK_wrapper.cpp', 'wrapper_common.cpp', ['-L' KinectSDKPathLib],
'-lKinect10', ['-I' KinectSDKPathInclude]);

Error in setup_kinectsdk (line 77)
makecmexfile4nid;

"

Please let me know if you can help at your earliest convenience.

Thank you

[sahar]

Hi Takashi,

Can the 'Simulink support for kinect' work on Windows 8 64bit??

[Takashi Chikamasa]

Concerning OpenNI, you need to use an older version of OpenNI (http://www.openni.org/openni-sdk/openni-sdk-history-2/). The latest OpenNI 2.1 beta is totally different from the older versions, thus, it does not work with Simulink support for Kinect.

[Mehwish]

Also, after I run the setup file, I'm supposed to get both files, sfun_nid.mexw32 and simulinkinfo.m, right? Because I've only been able to get simulinkinfo.m file so far.

[Mehwish]

Hi Takashi,

I'm still having the same problems.

1) Initially I was using the setup_kinectsdk.m file. When I run the .m file, I'm asked to specify the location of the Kinect installers. I couldn't solve the problem even when I used C:\Program Files\Microsoft SDKs\Kinect\v1.6

2) I know my Kinect SDK was installed properly. But just incase, I now tried with setup_openni.m file. OpenNI is installed correctly, as verified by Matlab as 'ok' and the camera is capturing images correctly when used as a webcam. But I'm experiencing the same errors as before. Problem with directory or path...

Do I have to change the code for any of the files to make it work for me?

>> dir1 = 'C:\Program Files\OpenNI';
>> setup_openni(dir1);
### Verifying OpenNI: OK
### Confirming Simulink Support for Kinect installations with OpenNI...
### Setting MATLAB path to .\Lib and .\Lib\doc_en directories.
Warning: Name is nonexistent or not a directory: C:\Users\EDP\Documents\MATLAB\Lib
> In path at 110
In addpath at 87
In setup_openni at 78
Warning: Name is nonexistent or not a directory:
C:\Users\EDP\Documents\MATLAB\Lib\doc_en
> In path at 110
In addpath at 87
In setup_openni at 79
### Generating C:\Users\EDP\Documents\MATLAB\Lib\simulinkfornidinfo.m
Error using fclose
Invalid file identifier. Use fopen to generate a valid file identifier.

Error in setup_openni (line 94)
fclose(fid);

[Takashi Chikamasa]

OK. You specified the wrong path for
SIMULINKFORNIDKINECTSDKPATH = 'C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\slkinect\Lib';

It has to be the path to MS Kinect SDK (not Simulink support for Kinect).
In my case,
SIMULINKFORNIDKINECTSDKPATH = 'C:\Program Files\Microsoft SDKs\Kinect\v1.6';

Please check the MS Kinect SDK could be installed properly.

[Mehwish]

However, I'm getting the following error:

>> setup_kinectsdk
### Confirming Simulink Support for Kinect installations with Kinect SDK...
### Setting MATLAB path to .\Lib and .\Lib\doc_en directories.
Warning: Name is nonexistent or not a directory:
C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\Lib
> In path at 110
In addpath at 87
In setup_kinectsdk at 54
Warning: Name is nonexistent or not a directory:
C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\Lib\doc_en
> In path at 110
In addpath at 87
In setup_kinectsdk at 55
### Generating C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\Lib\simulinkfornidinfo.m
Error using fclose
Invalid file identifier. Use fopen to generate a valid file identifier.

Error in setup_kinectsdk (line 68)
fclose(fid);

[Mehwish]

Hi Takashi,

I checked that file simulinkfornidinfo.m. And yes it says the following:

% This file is automatically generated by setup_kinectsdk.m
SIMULINKFORNIDDRIVER = 'KINECT_SDK';
SIMULINKFORNIDKINECTSDKPATH = 'C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\slkinect\Lib';

[Takashi Chikamasa]

Hi Mehwish,
Could you check the simulinkfornidinfo.m? if it was generated correctly, the file constains something like following:

% This file is automatically generated by setup_kinectsdk.m
SIMULINKFORNIDDRIVER = 'KINECT_SDK';
SIMULINKFORNIDKINECTSDKPATH = 'C:\Program Files\Microsoft SDKs\Kinect\v1.6';

[Mehwish]

Hi, I'm following the steps in the Simulink_Support_for_Kinect_En.pdf. I'm stuck on Step 5, which states to Run slkinect/setup_kinectsdk.

If everything is installed properly (which I'm sure it is), sfun_nid.mexw32/.mexw64 and simulinkfornidinfo.m should be generated.

However, following is the error I'm getting.

>> setup_kinectsdk
### Confirming Simulink Support for Kinect installations with Kinect SDK...
### Setting MATLAB path to .\Lib and .\Lib\doc_en directories.
### Generating C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\slkinect\Lib\simulinkfornidinfo.m
Error using setup_kinectsdk (line 69)
### ERROR: Failed to generate
C:\Users\EDP\Documents\MATLAB\simulink_support_for_kinect_0.6.0\slkinect\Lib\simulinkfornidinfo.m

[Mehwish]

 

[Samuel]

Hi Takashi,

Thanks for your quick answer. OpenNI and NITE do work on OSX with the PrimeSensor module for OSX (unstable). I was thinking that the Simulink interface should work too then. Thanks for the great work!

Sam

[Jiri Zemanek]

Hi Takashi,

thank you for the advice it was a stupid mistake :(. Anyway, you did a great job with this blockset! Thanks for that.

Jiri

[Takashi Chikamasa]

Hi Samuel,

>I'm trying the latest version with an Asus Xtion >Pro Live, but the NID Skeleton block always >causes MATLAB to crash.
ASUS Xtion PRO Live does not support for Mac OS X, only support for Win/Linux according to ASUS, so it is a known restriction by ASUS (neither OpenNI nor Simulink for NID)

[Samuel]

I'm trying the latest version with an Asus Xtion Pro Live, but the NID Skeleton block always causes MATLAB to crash. The depth and video blocks work fine however, and everything works fine with a Kinect.

I'm working with OpenNI on OSX.

[Takashi Chikamasa]

Hi Jiri,

You need to use the Kinect SDK 1.6 (not 1.0).

Takashi

[Jiri Zemanek]

Hi,

unfortunately, I'm not able to compile the cpp files. I have Kinect for Win (SDK 1.0), Matlab R2010b 64bit, MS VS 2010 running on Win7 64bit and after calling mex() I am getting a long list of errors (attached bellow). Do you have any idea what's wrong? Thanks.

kinectSDK_wrapper.cpp(54) : error C2143: syntax error : missing ';' before '*'
kinectSDK_wrapper.cpp(54) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
kinectSDK_wrapper.cpp(54) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
kinectSDK_wrapper.cpp(830) : error C2039: 'NuiGetCoordinateMapper' : is not a member of 'INuiSensor'
C:\Program Files\Microsoft SDKs\Kinect\v1.0\inc\NuiSensor.h(508) : see declaration of 'INuiSensor'
...

[Samuel]

 

[Takashi Chikamasa]

>Can we use this one for multiple kinects?
Sorry, Simulink for NID does not support multiple kinect devices. It requires huge changes in the code.

[Takashi Chikamasa]

>I am trying to output xyz coordinate depth data over >time. I would like to output these coordinates in real->time. Is this possible with this code? If so, how
>(exactly) can I implement this?
nid_cvst_point_cloud.mdl might be a good example to understand how to acquire xyz coordinates depth data. This model uses a Frame Rate
Display of CVST, but it has nothing with the depth data, so you can remove it and you can manipulate the xyz depth data in Simulink without CVST.

[Tharindu]

Can we use this one for multiple kinects?

[Mike Powell]

I am trying to output xyz coordinate depth data over time. I would like to output these coordinates in real-time. Is this possible with this code? If so, how (exactly) can I implement this?

What I have tried so far (among many other things):
1. Open the nid_depth.mdl block
2. Double click the NID Depth block and change output to XYZ.
3. Check "Real World Coordinates"
4. Double click on the selector. Change the number of input dimensions to 3.
5. Insert a simulink/Sinks/To Workspace block instead of the scope.
6. Change the time from inf to 1.5.
7. Double click the wire between the NID Depth block and the To Workspace block. Name the wire using a name without spaces.
7. Hit run.

When I do this, the output is a 1x1x46 timeseries. This is not what I am looking for.

To do what I am asking, do I need the computer vision system toolbox (which I do not currently have)?

[marchienne]

Thank you very much for your help. Your works are very useful !

[Takashi Chikamasa]

Hi marchienne,
You can create MATLAB Guide based viewer for any types of Kinect acquired data. I am going to bundle some example which uses this technique in the next update (maybe in October after MS released the SDK 2.0).

[marchienne]

Hi,

Is someone able to see the first examples? I have an error message that says "Unable to load block diagram 'vipsnks'". I saw that we had to use a toolbox named "Computer Vision System Toolbox" but there is a way to do it without this toolbox ?

Thank you for your help.

[Takashi Chikamasa]

Hi Mike,
Thank you for your information. I will add your tips to the documents in the next update.

[Mike Powell]

Actually, I just fixed the mex problem and it ran successfully using the recommedation listed at
http://www.mathworks.com/matlabcentral/answers/27759#answer_36115

"I had to copy "mspdb100.dll" from the "..\Microsoft Visual Studio 10.0\Common7\IDE\" to the "..\Microsoft Visual Studio 10.0\VC\bin\"."

by Stephen Yip.

[Mike Powell]

(and I wanted to add that I have a Simulink license)

Amanda

[Mike Powell]

Hi,

I have installed:
- Microsoft SDK 7.1
- Visual Studio C++ 2010 Express
- OpenNI Stable Build for Windows x86 v1.5.2.23 Redist Edition
- PrimeSense NITE Stable Build for Windows x86 v 1.5.2.21 Redist Edition
- PrimeSensor Module Stable Build x86 v 5.1.0.41
- Kinect for Windows SDK v1.5

However, when I open setup_kinectsdk.m in MATLAB and hit run, I get the following error:

C:\PROGRA~1\MATLAB\R2012A\BIN\MEX.PL: Error: Compile of 'sfun_nid.cpp' failed.

Error using mex (line 206)
Unable to complete successfully.

Error in makecmexfile4nid (line 62)
mex('sfun_nid.cpp', 'kinectSDK_wrapper.cpp', 'wrapper_common.cpp', ['-L' KinectSDKPathLib], '-lKinect10', ['-I'
KinectSDKPathInclude]);

Error in setup_kinectsdk (line 76)
makecmexfile4nid;

I have tried changing the compiler using mex -setup, but neither compiler works. The compiler options I tried were:

[1] Lcc-win32 C 2.4.1 in C:\PROGRA~1\MATLAB\R2012a\sys\lcc
[2] Microsoft Software Development Kit (SDK) 7.1 in C:\Program Files\Microsoft Visual Studio 10.0

Any assistance or suggestions would be appreciated.

Amanda

[Akhand Tripathy]

 

[Takashi Chikamasa]

I forgot to inform important thing about 0.5.1:
- Support OpenNI in MAX OS X! and Linux!

[Takashi Chikamasa]

Hi Francisco,

I assume you have Simulink license. Is this correct?

[Francisco]

Hello everyone. I am having the same problems as Jno navarro. Any solution to this problem?

c:\users\jno\desktop\kinect project\simulink_nid_0.4.1\slnid\lib\sfun_nid.h(16) : fatal error C1083: Cannot open include file: 'simstruc.h': No such file or directory

C:\PROGRA~1\MATLAB\R2010B\BIN\MEX.PL: Error: Compile of 'sfun_nid.cpp' failed.

??? Error using ==> mex at 208
Unable to complete successfully.

Error in ==> makecmexfile at 54
mex('sfun_nid.cpp', 'kinectSDK_wrapper.cpp', 'wrapper_common.cpp', ['-L'
KinectSDKPathLib], '-lKinect10', ['-I' KinectSDKPathInclude]);

Error in ==> setup_kinectsdk at 78
makecmexfile;

[Takashi Chikamasa]

What's new 0.4.2 in detail
-Support Skeleton tracking in the Kinect SDK v1.5 Near mode and Seated mode
-Support multiple person skeleton tracking (up to two persons)
-Support multi-instantiation of NID Device blocks in a model

[In Japanese]
-Kinect SDK v1.5の最新機能である、Near mode、Seated modeにおける骨格トラッキングに対応
-最大2人までの複数ユーザーの同時骨格トラッキングに対応
-同一NID Deviceブロックのモデル内複数配置に対応

[Takashi Chikamasa]

Hi jno,

Great to hear your success in your project. As far as I tested with a Kinect for Windows HW, depth and image with Near mode could work fine. So it sounds strange. In case of Near mode, I am sure that skeleton tracking does not work in v1.0 Kinect SDK. According to MS, they will release v1.5 in May and it seems to support skeleton tracking in Near mode. Anyways, your 3D scanner should be absolutely interesting for us. If you published some of your work as video such as at YouTube, please let me know.

[jno navarro]

Hi Takashi,
Thank you for your fast answers, my 3d scanner is working quite well, the only thing is that i am not able to use near mode, i am using kinect sdk last version and everything seems to work well except that whe i turn on near mode after using my application it says fail to use near mode and it doesnt work. Do you know why this may happen?
Thank you

[Takashi Chikamasa]

Hi jno,

>so the element 1,1 of the matrix in real
>world coordinates is in the case of the
>x matrix, the x position of the point in
>that pixel, is that correct?
In my experience with the point cloud sample. You assumption should be correct. The point cloud viewer is mapping image on each pixel onto real world XYZ coordinates at the same pixel.

[jno navarro]

Hi Takashi,
I have readed the help format, but what i uderstando is that the depth block gives me, in the case x/y/z, the x y and z position of each pixel, formed by a 640*480 matrix, so the element 1,1 of the matrix in real world coordinates is in the case of the x matrix, the x position of the point in that pixel, is that correct??

[Takashi Chikamasa]

Hi jno,

Please see the help of Depth block. You can find description about depth format.

[jno navarro]

Hi Takashi,
I found the error in another blog, there is a problem with my matlab and it is already fixed. Your program runs great, can you tell me in which format you get the points of depth?

[Takashi Chikamasa]

Hi jno,

Did you install Simulink on your PC? Simuilnk for NID is for Simulink, not for MATLAB.

[jno navarro]

Hi everyone,
I have Matlab 7.11.0 (r2010b) in a win 7 64 bits. Each time I try to compile the archives using vs2010 professional I get the following error:
sfun_nid.cpp
c:\users\jno\desktop\kinect project\simulink_nid_0.4.1\slnid\lib\sfun_nid.h(16) : fatal error C1083: Cannot open include file: 'simstruc.h': No such file or directory

C:\PROGRA~1\MATLAB\R2010B\BIN\MEX.PL: Error: Compile of 'sfun_nid.cpp' failed.

??? Error using ==> mex at 208
Unable to complete successfully.

Error in ==> makecmexfile at 54
mex('sfun_nid.cpp', 'kinectSDK_wrapper.cpp', 'wrapper_common.cpp', ['-L'
KinectSDKPathLib], '-lKinect10', ['-I' KinectSDKPathInclude]);

Error in ==> setup_kinectsdk at 78
makecmexfile;

This is really urgent, I am trying to use my kinect to generate 3d models through matlab but I am stuck with that error, I would appreciate anyway to get images through matlab
Thank you in advance

[Tharindu]

Great code. thanks for sharing.
and thanks for the quick responses to the questions

[Takashi Chikamasa]

Hi lev,

I think ASUS Xtion does not have an accelerator sensor, so we can't get the tilt angles.

[lev]

Hi Takashi
Can we use ASUS Xtion to get the tilt angles instead of Kinect.

[Takashi Chikamasa]

Hi Michael and lev,
The latest version (0.4.1) supports your requests. It will be published soon.

[lev]

Hi Takashi
please add an outport on IMAQ to get the tilt angles.

[Michael]

Thank you sir. If you could do that, it would be much appreciated.

[Takashi Chikamasa]

Hi Michael,

>Can you advise me on an easy way to
>access the vector data pertaining to the
>skeletal joint points?
point_cloud_viewer.m might be useful to understand how to access the data easily. I am sure both of eML and Simulink, you can access the data pretty easy. I also consider adding a demo based on your question.

[Takashi Chikamasa]

Hi Lev,

>if you can provide the heading and tilt angles.
The Kinect tilt angle can be acquired by the Kinect SDK, but heading angle is not supported by the SDK. If the tilt angle is sufficient, I can add an outport on IMAQ to get the tilt angle.

[lev]

 

[Michael]

Can you advise me on an easy way to access the vector data pertaining to the skeletal joint points?

Pretty much what I am trying to do is use the joint 3d data to control servos, making a robotic arm mimic your movements.

[lev]

Hi Takashi,
Can I obtain the accelerometer data using this Simulink, I deeply appreciate if you can provide the heading and tilt angles.

[Takashi Chikamasa]

A new feature of Kinect for Windows hardware "Near mode" also worked fine with the latest Simulink for NID.

[Japanese]
Kinect for Windowsハードウェアの新機能である"Near Mode"も正常に動作することを確認しました。

[Takashi Chikamasa]

[Japanese]
ver.0.4.0から、マイクロソフトのKinect for Windows SDK v1.0にも対応しました。DSP System Toolboxを使用することで、Kinect for Windowsの4chマイクロフォンアレイ生データもSimulink上で取得することができます。

[Emanuele Ruffaldi]

Hi Takashi,

I am referring to the ONI files that can be recorded with NiViewer provided with OpenNI. These ONI files contain the various Kinect streams (depth and/or image). Using this patch it is possible to keep the same Simulink model and select to get data from a recorded ONI file or from the real sensor. It is useful for testing a model with consistent input (e.g. different users). The patch works by adding a FILE_BLOCK (7) to the S-Function dispatcher, and selecting it if the filename parameter of the mask is not empty.

It is even possible to have NiViewer recording data while using the real Kinect from Simulink.

Best,
Emanuele

[Takashi Chikamasa]

Hi Emanuele,

Thank you for your post. Next update will support both of 32bit/64bit version of MATLAB. Concerning data recording, there are several options in Simulink:
1. To/From Workspace blocks
If you also have a Computer Vision System Toolbox, you can use
2. To/From Multimedia File blocks
3. Video To/From Workspace blocks

[Emanuele Ruffaldi]

I've made an update with two changes:
- support for MATLAB 64bit
- support for using recorded ONI files as input

There is now a new block for file input.
Another one allows mixed situation: if the filename is empty it uses the real-time input otherwise the recorded file.

Thanks for this greate package

Source code: http://dl.dropbox.com/u/146279/simulink_nid_0.3.1file.zip

[Takashi Chikamasa]

[English]
ver.0.3.1 supports QVGA(320x240): 60FPS IMAQ for ASUS Xtion PRO series devices.

[Japanese]
ver.0.3.1では、ASUS Xtion PROシリーズデバイスを対象に、QVGA(320x240): 60FPSのIMAQをサポートしました。

[Takashi Chikamasa]

%% Description in Japanese
Simulink for NID は、IMAQ, Image, IR, Depth, Motion および Skeletonブロックから構成されており、OpenNIが対応しているデバイスからのリアルタイムデータ取得が可能です。Simulink for NIDの使用に際しては、32bitバージョンのR2010b以降のMATLAB製品をWindows7 32bit/64bit環境にインストールする必要があります。また、次のサードパーティソフトウェアが環境構築に必要になります:
- Microsoft Visual Studio 2010 (VC++) Express Edition
- OpenNI SDK
- PrimeSense NITE
- OpenNIに対応したNID向けデバイスドライバ
※詳細については、ZIPファイルに同梱されている日本語readme_ja.textおよび各ブロックの日本語ヘルプを参照してください。

[推奨MATLAB製品]
- Computer Vision System Toolbox
- Simulink Coder