Semi-Structured Intelligent Bin Picking for UR5e Using YOLO and PCA-Based Object Detection
This example shows detailed workflow for intelligent bin picking using Universal Robots UR5e cobot. The MATLAB project provided with this example consists of the Initialize, Perception, Motion Planning, and Integration modules (project folders) to create a complete bin picking workflow.
Open the Project
To get started, open the example live script and access the supporting files by either clicking Open Live Script in the documentation or using the
Then, open the Simulink™ project file.
prj = openProject('BinPickingUR5eHW/BinPickingApplicationWithUR5eHardware.prj');
Bin Picking Workflow
This diagram shows the operations involved in the one complete execution cycle for the cobot bin picking application.
In this example, we are targeting the semi-structured bin picking, which is a classification based on the bin complexity.
On a high level, this task can be divided into two major modules:
Vision processing / Perception module
Motion / Trajectory planning module
Vision Processing or Perception module
This workflow can be further divided into two areas:
Object detection using RGB data, which is object segmentation based on RGB and depth (RGBD) data using deep learning
Object pose estimation using 3D point cloud data, which can be used to estimate the pose of the identified object for motion planning
This diagram shows the steps involved in the object pose estimation using YOLOv4 and PCA-based algorithm.
The perception process involves two main steps:
Training and validation - Training the RGB-based object detection network (YOLO v4) and its validation against the test dataset.
Online object detection - Pose estimation using real-time raw RGBD data using a pre-trained YOLO v4 network and Principle Component Analysis (PCA).
Motion Planning Module
This workflow can be divided into two areas:
Smart motion planning, which picks the logic for motion planning from the detected parts
Goal execution, which executes trajectory planning for pick and place operation using UR5e cobot
Using the Live Script in the
Motion Planning folder available in the MATLAB project of this example, you generate a MEX function using C/C++ code generation for the motion planning of the bin picking application. The motion planning algorithm as explained in Motion Planning and RigidBodyTree Simulation of UR5e for Bin Picking Using manipulatorRRT Algorithm is used to create a MEX function.
Generating a MEX function using C/C++ code generation helps you to reduce the computation time, thereby reducing the pick and place cycle time.
For more information on MEX function creation for the manipulatorRRT algorithm-based planner, refer to Generate Code for Manipulator Motion Planning in Perceived Environment example.
Also, refer to this example to know more about how to generate MEX function to accelerate your MATLAB program execution.
This example uses:
Robotics System Toolbox
Computer Vision Toolbox
Deep Learning Toolbox
Image Processing Toolbox
Statistics and Machine Learning Toolbox
MATLAB Coder (Required if you want to use MEX function for motion planner)
Robotics System Toolbox Support Package for Universal Robots UR Series Manipulators
Computer Vision Toolbox Model for YOLO v4 Object Detection support package (Required if you want to train a detector model)
Universal Robots UR5e
Robotiq EPick vaccum suction gripper
Interface Used for Universal Robots UR5e
The functionality from Robotics System Toolbox Support Package for Universal Robots UR Series Manipulators is used for trajectory and joint control of the Universal Robots UR5e hardware. The support package offers
universalrobot object and its associated functions to enable control over the ROS interface.
Interface Used for Robotiq Epick suction gripper
Most of the real-world cobot applications involve operations such as pick and place, dispensing, and so on, and this requires attaching and using any external end-effector tool according to the application. The
universalrobot offers one useful method called
handBackControl which allows you to perform end-effector operations using standard URCaps along with the ROS external control.
Interface Used for Intel® RealSense™ D415
For perception purposes, Intel® RealSense™ D415 camera depth module is used in this example. The MATLAB connection is established over the ROS using the IntelRealSense ROS driver.
The dataset used for the training of the YOLOv4 deep learning network has been created using this sensor.
You can follow the detailed steps given for the installation of the required ROS drivers for the connection.
Physical Setup Used in this Example
For the demonstration of the intelligent bin picking workflow using the Universal Robots UR5e, this example uses a cuboid shape whiteboard eraser as the object and a rectangular bin.
The dimensions of the cuboid and bin are provided in the
initializeParametersForBinPickingHardware.m script in the Initialize folder of the project.
RigidBodyTree and Actual Environment setup
In this example, we will create a RigidBodyTree environment for motion planning.
This flowchart shows the complete workflow that uses the Live Scripts available in Perception, Motion Planning, and Integration folders of the MATLAB project.
The initialization script
initializeParametersForBinPickingHardware.m runs automatically when you open this MATLAB project. This script defines some of the important parameters used in the perception, motion planning, and integration project module. You can find this script inside the Initialize folder or you can run the below command to open the script. Change the parameters if you are running with some different setup acccordingly.
Open Perception Module Script
Run this command to open the script of the perception workflow. This script covers the full perception pipeline from training to the object pose estimation workflow.
Open Motion Planning Module Script
Run this command to open the script for the motion planning workflow. This script covers the RigidBodyTree simulation workflow and MEX function generation steps for the motion planning module.
Open Integration Module Script
Run this command to open the main script of the integrated workflow. This script shows how to use perception and motion planning modules for creating a full bin picking application workflow using the Universal Robots UR5e hardware.