Simulink for Natural Interaction Device (NID)

Simulink for NID is a set of Simulink blocks interfaced with OpenNI and Microsoft Kinect SDK.

Description

Simulink for NID consists of IMAQ, Image, IR, Depth, Motion and Skeleton blocks to achieve real-time image data acquisition from NID such as ASUS Xtion PRO LIVE or Microsoft Kinect for Windows under Simulink simulation environment.
It works with 32bit/64bit version of MATLAB R2010b or later in Windows7 32bit/64bit and it requires the following 3rd party software:

[C MEX Compiler]
- Microsoft Visual Studio 2010 Express Edition (VC++)
- Microsoft Windows SDK 7.1

[To use OpenNI (http://www.openni.org/)]
- OpenNI
- PrimeSense NITE
- PrimeSensor Module

[To use Microsoft Kinect SDK (http://www.microsoft.com/en-us/kinectforwindows/)]
- Microsoft Kinect for Windows SDK v1.5

For more detailed information, please refer to document in the zip file and help of the blocks.

[Optional MATLAB products]
- Computer Vision System Toolbox (highly recommended)
- DSP System Toolbox (to acquire Kinect Microphone Array data)
- Simulink Coder (to generate *.exe file)

Acknowledgements

The author wishes to acknowledge the following in the creation of this submission:
Kinect Matlab, Kinect Microsoft SDK

• Required Products: Simulink
• MATLAB release: MATLAB 7.11 (2010b)

[Takashi Chikamasa]

%% Description in Japanese
Simulink for NID は、IMAQ, Image, IR, Depth, Motion および Skeletonブロックから構成されており、OpenNIが対応しているデバイスからのリアルタイムデータ取得が可能です。Simulink for NIDの使用に際しては、32bitバージョンのR2010b以降のMATLAB製品をWindows7 32bit/64bit環境にインストールする必要があります。また、次のサードパーティソフトウェアが環境構築に必要になります:
- Microsoft Visual Studio 2010 (VC++) Express Edition
- OpenNI SDK
- PrimeSense NITE
- OpenNIに対応したNID向けデバイスドライバ
※詳細については、ZIPファイルに同梱されている日本語readme_ja.textおよび各ブロックの日本語ヘルプを参照してください。

[推奨MATLAB製品]
- Computer Vision System Toolbox
- Simulink Coder

[Takashi Chikamasa]

[English]
ver.0.3.1 supports QVGA(320x240): 60FPS IMAQ for ASUS Xtion PRO series devices.

[Japanese]
ver.0.3.1では、ASUS Xtion PROシリーズデバイスを対象に、QVGA(320x240): 60FPSのIMAQをサポートしました。

[Emanuele Ruffaldi]

I've made an update with two changes:
- support for MATLAB 64bit
- support for using recorded ONI files as input

There is now a new block for file input.
Another one allows mixed situation: if the filename is empty it uses the real-time input otherwise the recorded file.

Thanks for this greate package

Source code: http://dl.dropbox.com/u/146279/simulink_nid_0.3.1file.zip

[Takashi Chikamasa]

Hi Emanuele,

Thank you for your post. Next update will support both of 32bit/64bit version of MATLAB. Concerning data recording, there are several options in Simulink:
1. To/From Workspace blocks
If you also have a Computer Vision System Toolbox, you can use
2. To/From Multimedia File blocks
3. Video To/From Workspace blocks

[Emanuele Ruffaldi]

Hi Takashi,

I am referring to the ONI files that can be recorded with NiViewer provided with OpenNI. These ONI files contain the various Kinect streams (depth and/or image). Using this patch it is possible to keep the same Simulink model and select to get data from a recorded ONI file or from the real sensor. It is useful for testing a model with consistent input (e.g. different users). The patch works by adding a FILE_BLOCK (7) to the S-Function dispatcher, and selecting it if the filename parameter of the mask is not empty.

It is even possible to have NiViewer recording data while using the real Kinect from Simulink.

Best,
Emanuele

[Takashi Chikamasa]

[Japanese]
ver.0.4.0から、マイクロソフトのKinect for Windows SDK v1.0にも対応しました。DSP System Toolboxを使用することで、Kinect for Windowsの4chマイクロフォンアレイ生データもSimulink上で取得することができます。

[Takashi Chikamasa]

A new feature of Kinect for Windows hardware "Near mode" also worked fine with the latest Simulink for NID.

[Japanese]
Kinect for Windowsハードウェアの新機能である"Near Mode"も正常に動作することを確認しました。

[lev]

Hi Takashi,
Can I obtain the accelerometer data using this Simulink, I deeply appreciate if you can provide the heading and tilt angles.

[Michael]

Can you advise me on an easy way to access the vector data pertaining to the skeletal joint points?

Pretty much what I am trying to do is use the joint 3d data to control servos, making a robotic arm mimic your movements.

[lev]

 

[Takashi Chikamasa]

Hi Lev,

>if you can provide the heading and tilt angles.
The Kinect tilt angle can be acquired by the Kinect SDK, but heading angle is not supported by the SDK. If the tilt angle is sufficient, I can add an outport on IMAQ to get the tilt angle.

[Takashi Chikamasa]

Hi Michael,

>Can you advise me on an easy way to
>access the vector data pertaining to the
>skeletal joint points?
point_cloud_viewer.m might be useful to understand how to access the data easily. I am sure both of eML and Simulink, you can access the data pretty easy. I also consider adding a demo based on your question.

[Michael]

Thank you sir. If you could do that, it would be much appreciated.

[lev]

Hi Takashi
please add an outport on IMAQ to get the tilt angles.

[Takashi Chikamasa]

Hi Michael and lev,
The latest version (0.4.1) supports your requests. It will be published soon.

[lev]

Hi Takashi
Can we use ASUS Xtion to get the tilt angles instead of Kinect.

[Takashi Chikamasa]

Hi lev,

I think ASUS Xtion does not have an accelerator sensor, so we can't get the tilt angles.

[Tharindu]

Great code. thanks for sharing.
and thanks for the quick responses to the questions

[jno navarro]

Hi everyone,
I have Matlab 7.11.0 (r2010b) in a win 7 64 bits. Each time I try to compile the archives using vs2010 professional I get the following error:
sfun_nid.cpp
c:\users\jno\desktop\kinect project\simulink_nid_0.4.1\slnid\lib\sfun_nid.h(16) : fatal error C1083: Cannot open include file: 'simstruc.h': No such file or directory
 
  C:\PROGRA~1\MATLAB\R2010B\BIN\MEX.PL: Error: Compile of 'sfun_nid.cpp' failed.
 
??? Error using ==> mex at 208
Unable to complete successfully.

Error in ==> makecmexfile at 54
    mex('sfun_nid.cpp', 'kinectSDK_wrapper.cpp', 'wrapper_common.cpp', ['-L'
    KinectSDKPathLib], '-lKinect10', ['-I' KinectSDKPathInclude]);

Error in ==> setup_kinectsdk at 78
makecmexfile;
  

This is really urgent, I am trying to use my kinect to generate 3d models through matlab but I am stuck with that error, I would appreciate anyway to get images through matlab
Thank you in advance

[Takashi Chikamasa]

Hi jno,

Did you install Simulink on your PC? Simuilnk for NID is for Simulink, not for MATLAB.

[jno navarro]

Hi Takashi,
I found the error in another blog, there is a problem with my matlab and it is already fixed. Your program runs great, can you tell me in which format you get the points of depth?

[Takashi Chikamasa]

Hi jno,

Please see the help of Depth block. You can find description about depth format.

[jno navarro]

Hi Takashi,
I have readed the help format, but what i uderstando is that the depth block gives me, in the case x/y/z, the x y and z position of each pixel, formed by a 640*480 matrix, so the element 1,1 of the matrix in real world coordinates is in the case of the x matrix, the x position of the point in that pixel, is that correct??

[Takashi Chikamasa]

Hi jno,

>so the element 1,1 of the matrix in real
>world coordinates is in the case of the
>x matrix, the x position of the point in
>that pixel, is that correct?
In my experience with the point cloud sample. You assumption should be correct. The point cloud viewer is mapping image on each pixel onto real world XYZ coordinates at the same pixel.

[jno navarro]

Hi Takashi,
Thank you for your fast answers, my 3d scanner is working quite well, the only thing is that i am not able to use near mode, i am using kinect sdk last version and everything seems to work well except that whe i turn on near mode after using my application it says fail to use near mode and it doesnt work. Do you know why this may happen?
Thank you

[Takashi Chikamasa]

Hi jno,

Great to hear your success in your project. As far as I tested with a Kinect for Windows HW, depth and image with Near mode could work fine. So it sounds strange. In case of Near mode, I am sure that skeleton tracking does not work in v1.0 Kinect SDK. According to MS, they will release v1.5 in May and it seems to support skeleton tracking in Near mode. Anyways, your 3D scanner should be absolutely interesting for us. If you published some of your work as video such as at YouTube, please let me know.

[Takashi Chikamasa]

What's new 0.4.2 in detail
-Support Skeleton tracking in the Kinect SDK v1.5 Near mode and Seated mode
-Support multiple person skeleton tracking (up to two persons)
-Support multi-instantiation of NID Device blocks in a model

[In Japanese]
-Kinect SDK v1.5の最新機能である、Near mode、Seated modeにおける骨格トラッキングに対応
-最大2人までの複数ユーザーの同時骨格トラッキングに対応
-同一NID Deviceブロックのモデル内複数配置に対応