Programming Robots with ROS and ROS2 using MATLAB and Simulink

Hello, everyone. Thank you for joining, and welcome to this webinar on programming robots with ROS using MATLAB and Simulink.

Hi. My name is YJ Lim, technical product manager for robotics and autonomous systems at The MathWorks. I am with Murat Belge. He is a development lead for ROS toolbox at The MathWorks.

Industrial manufacturing paradigms are shifting from mass production to mass customization. Autonomous robotics systems become more complex. After the first industrial robot deployed in 1961, over 3 million robots are in the manufacturing floor. Industrial robotics systems were oriented toward mass production of products in factories, where robots were destined to carry out repetitive and individual works in the controlled environment.

The current tendency is toward robotic systems that must be capable of solving the problem in more complex and a less controlled environment. Collaborative robots, or cobots, are now becoming available for more autonomous tasks, because it is designed to be more versatile. Cobots use sensor input from the environment to control and make decisions.

To this end, robotic systems are needed that are more autonomous and intelligent. AI-enabled robots equipped with the autonomous algorithms and perceptions are getting started during the past few years. Now, robots collect more data from the environment and learn from interaction. Over the next few years, we will see the continuous growth of the AI-based robots.

Autonomous systems may have a mix of multiple synchronous processing for more precise control, as well as an asynchronous interface. Autonomous system control may require parallel or distributed processing, running on multicore, or GPU for high-resolution and multirate sensor and the perception processing. It produces coordinated motion of multi-degree of  freedom actuators in autonomous systems.

Several robotics software frameworks have been appeared that propose some solutions to design, and develop the distributed system of autonomous systems. ROS has been used in the development of large autonomous systems. ROS stands for Robot Operating System, but not in the operating system. It is a middleware for autonomous distributed system. At the core, ROS has well established a tool for inter-process messaging and distributed computing. At a high level, ROS has a packaging and build  management system that is not only used by the core ROS developer, but also by the user community.

ROS continues to grow in popularity. And the ROS user community is very important, because they are making application package drivers to talk to the hardware, interface to other software packages, and the robotic-specic tools for tasks like perception, motion planning, and so on. After gathering requirements from industrial participants for years. ROS 2, a new version of ROS 1, has been released for research and production in mind.

Some key new things are support for  small embedded and real-time systems and quality-of-service in messaging, which is enabled by data distribution  service, DDS, standard that is ubiquitous in the industry. The community is gradually migrating to ROS currently.  Autonomous systems can be efficiently developed by utilizing ROS.

So today, I will begin by introducing ROS capability from MATLAB and Simulink. Then I will share some user cases from our customers. Then I will discuss about ROS and ROS 2 workflow with MATLAB and Simulink. Then Murat will show you some ROS application examples using MATLAB and Simulink.

Let me dive into ROS capability from MATLAB and Simulink. You can connect ROS and ROS 2 from MATLAB and Simulink for design and development of robotics and autonomous systems. With a live of connection from MATLAB and Simulink to an existing ROS network, you can explore available topics, a service, and an action. This also allows you to send commands to, and receive the data from, any simulator, hardware or software node on the ROS network. MATLAB and Simulink also provides utilities to work with some common sensor data type, such as laser scan, images, and 3-D point clouds.

If you are using recorded data from a rosbag file, our toolbox allows you to read these files for post-processing or playback data as a simulation input. After this initial testing, you can automatically generate C++ based executable ROS nodes from your design. These generated node can be integrated with ROS and ROS 2 build the system, so they can be run on your target system with no  dependence on MATLAB and Simulink.

So what ROS functionality do we offer today? As you can see, there is support for both MATLAB and Simulink workflow, although they are not quite the same. A key difference is code generation. It only work from  Simulink models today, although we have some workaround to do and future plans. ROS 2 interface is fairly lightweight, but it covers the most common user cases. You are able to do pub and sub with the ROS 2 from both MATLAB and Simulink. And you can also generate standalone ROS 2 nodes from Simulink models. With both ROS 1 and ROS 2, you can generate your own custom message.

In R2020b release, we support  both ROS 1 and ROS 2 on all major operating systems. At the moment of 2020b release, we supported melodic LTS ROS 1 and  dashing LTS for ROS 2, respectively. We replicated the ROS commander line tools, so if you are already familiar with open-source ROS, you could adopt MATLAB API functions easily.

In future releases, ROS node generation from MATLAB is our next priority.  With this feature, you can run multiple nodes, coded in MATLAB in parallel . We are also working to meet the parity between ROS 1 and ROS 2 support. We also have a plan to support ROS 1 noedic, which is the final version of ROS1 LTS.

So now let me introduce a couple of user stories from our customers. Engineers at Clearpath Robotics use MATLAB to speed up the algorithm development for the company's OTTO line and to advance robotic research. Teams from Clearpath needed to analyze and visualize large and complex data sets. They wanted a tool that complements  ROS capabilities, making it easier to process and analyze ROS data, including live streaming data and data captured in ROS log files.

Clearpath engineers use MATLAB with ROS capability to prototype algorithms and analyze and visualize data for their robotic research and development.

This is another user story from Voyage. Voyage is developing a controller for a self-driving car to follow a target velocity and maintain a safe distance from obstacles. Engineers at Voyage use Simulink to design a longitudinal model predictive controller and tune the parameters based on experimental data imported into MATLAB using ROS support. And they deployed the controller as a ROS node.

Let me talk about ROS and ROS 2 workflow with MATLAB and Simulink. ROS is an enabling technology to develop complex intelligence systems. With given system design requirements, ROS provides a common framework to design, simulate, prototype robot applications. With ROS, you can analyze log data, design algorithms with various handwritten codes, and visualize and debug your results. You may need to rewrite source code for production though.

By connecting MATLAB and Simulink with ROS, it allows you to perform system-level simulation with requirement traceability. It also provides rapid and iterative design process with many of the tools MATLAB and Simulink has for autonomous systems. You can also generate certifiable code for production.

So, now, let's see how we bridge ROS with MATLAB and Simulink. In the right-hand side, this is how traditional ROS-enabled systems might look like. There is a host of ROS-enabled simulator and hardware. And through the ROS network, they can communicate with a node that would enable autonomy, visualization, and so on. In the left-hand side, MATLAB and Simulink offer a tool for autonomous system development. For autonomoussystem development, the system should have functional components of perception, planning, and control. And now, we have a tight bridge to ROS from MATLAB and Simulink.

There are basically three workflows. The first is data analysis and playback. You log the data from your simulation and hardware, and then you want to bring them into the MATLAB and Simulink environment for offline design task. Next is the desktop simulation. This will be a live connection between ROS and an instance of MATLAB for algorithm development. Finally, deployment-- the same implementation you may have used for prototyping can be automatically generated as a standalone ROS node that we no longer require the MATLAB to run.

I want to point out, beyond just the bridging ROS with MATLAB and Simulink, you can complement a traditional ROS solution with MATLAB and Simulink tooling.

In offline data analysis workflow, the typical user case is to import data from a simulator or hardware to better understand the target system, or maybe to prototype algorithms. For example, you can read a batch of images for labeling and training a neural network.

You can extract the specific topic from time stamped data to synchronize and visualize them. The ability to read a rosbag file and to work with specialized ROS message types for sensors allows you to bring all your design data into MATLAB and Simulink without the need for a lot of pre-processing code.

For the sign-detection example, you can use computer vision and deep learning  to automatically learn image features.

Here, you can label image topics for detecting signs. You can use laser scan and odometry data imported from rosbag file to create a map of environment. In this example, you can use SLAM Map Builder app fromNavigation Toolbox to map the environment using lidar scan topic.

The second workflow is ROS network and communication. Your autonomous algorithm can be prototyped or evaluated by connecting the ROS network to the external simulator or hardware.

Here, we are showing a sign following robot again, running on Gazebo simulator with a Simulink in the loop. Sign-detection algorithm is running in Simulink. It may be either simple image processing with a color detection, or more advanced deep learning algorithm that you saw in the previous slide. and the velocity command will be published through ROS network to control robot. We have a virtual machine image that you can download. It has a ROS 1, ROS 2 installation there with several examples you can try out. MATLAB and Simulink could make desktop simulation easier with ROS, together with many other design tools, to test and verify new robotics algorithms.

Let's just take a look at a simple MATLAB and ROS interface workflow. Within MATLAB, you can create a ROS network, and participate in an existing ROS network. You can then collect the data, send and receive the ROS message. Your algorithm can be processed to schedule the execution of a MATLAB code efficiently.  And publish your command to control your robot. Then you can visualize your simulation result.

Similarly, in the Simulink, you need to set up with some configurations to establish the connection with ROS. And then you have a control loop, which is continuously sensing the data over the network, doing some kind of processing of algorithms, then sending out the command or control signal back through the ROS network.

One thing to note here-- you can see these blocks that I highlighted, which include ROS in it. These blocks are ROS blocks, coming from the ROS toolbox to read and data, subscribe the topic, as well as create the message, and publish the ROS topic. These blocks allow you to communicate with ROS without writing any codes.

Finally, node generation for deployment. The verification is an important task. Your robot algorithm needs an evaluation with a simulation or the robot hardware. You can manually port your algorithm to work on your target system. Alternatively, you can automatically, do this process with ROS node generation. This can speed up the development process,and remove manual implementation error.

Once you've generated a ROS node from Simulink, there are several ways you can work with it. For example, in Simulink, you can use an external mode to tune the generated ROS node in situ, even though the code itself is running on the target computer. It can be a good debugging tool while you are still designing your algorithm to tune the parameters and visualize data directly in the Simulink model.

Now I would like to turn it to Murat to take you through some ROS 1 and ROS 2 application examples with MATLAB and Simulink.

Thanks, YJ. Hello, everybody. This is Murat Belge, product lead for ROS Toolbox. Now that we have a good idea about the capabilities offered by ROS Toolbox. Let's get into a couple of application examples.

The first application example I am going to show is this sign-following robot. The robot, a TurtleBot2in this case, will identify posed signs and follow them to a given destination. We will develop the sign-posting navigation algorithm in Simulink and utilize a ROS2 network for communication. We  will use Gazebo to simulate the TurtleBot 3 and  the environment. The Simulink model controlling the robot will receive images and odometry data published from Turtlebot 3, compute velocity comments, and send them back to Turtlebot 3.

Note that this setup illustrates a powerful concept. We can develop our navigation algorithm, test them, verify it entirely in simulation environment. Since we are moving in a virtual environment created in Gazebo, we are allowed to make mistakes. For example, the robot can hit the walls. This might be catastrophic in real life, but it is an opportunity to improve the navigation algorithm in simulation.

Once we are satisfied that the algorithm works as intended, we can easily deploy  the Simulink model in a standalone ROS node to run on real hardware. Before you move on to detailed steps for setting up and running the application example, I'm going to quickly show the final result. As you can see, the robot follows directional signs placed in the room until it identifies a stop sign.

Now let's get into details workflow steps. What you see on the right-hand side is the Gazebo simulator running in an Ubuntu VM. This VM is shipped as an add-on to ROS Toolbox. In the main window of the Gazebo simulator, you can see the environment and the TurtleBot3 robot.

Now in MATLAB, I'm going to set up an environment variable called ROS_DOMain_ID. ROS 2 uses DDS as a transport layer. DDS supports physical segmentation of the network based on a number called domain ID. In order to interact with one another, two ROS 2 nodes need to be on the same DDS domain.

In this example, we set up the Gazebo simulator to create a DDS domain with ID 25. First, let's create a ROS 2 node in MATLAB. Note that we specify 25 as the domain ID. We can now talk to the robot running in Gazebo through this MATLAB node and dynamically discover publishers, subscribers, and more using ROS MATLAB interface. You can get a list of the available topics in the ROS 2 network using the ROS 2 topic list command in MATLAB.

Let's dig into the odometry topic. First, we create a subscriber to receive messages from /odom topic. We can then access the latest message, and display to xyz coordinates of the robot's position. You can send comments to the robot as well. To do this, we create a velocity publisher. The publisher also provides an empty message of the type geometry messages. We can set the angular velocity to 0.75 radians per second, and watch our robots rotate around the z-axis. As you can see, robots started to move in the Gazebo simulator.

To stop the robot, just reset the angular velocity to zero. Using MATLAB command line user interface, you can interactively work with the robot to get a feel for the messages, visualize, data, and do a quick and dirty prototype for the navigation.

Now, let's take a look at the Simulink side of the things. As a part of the 2020b release, ROS Toolbox supports deploying a Simulink model as a standalone C++ ROS 2 node. Since one of the objectives of this application example is to deploy the navigational document as  a standalone ROS node, we will be using Simulink to develope the algorithm. On the left is our Simulink model. It uses the camera images and odometry information as inputs.

We have an image-processing subsystem that identifies and tracks the signs. Here, blue sign is for left turn, green sign is for right turn, and red sign is for stop. In this example, we use simple blob detection to extract color thus the direction information. Let’s run the Simulink model. As you can see, the last two values are being constantly updated. You can also go to the Linux terminal and view messages directly. First, you perform setup tasks required to receive messages from the ROS network. Then we can read the detected blob information published from Simulink model directly on the terminal.

You can easily analyze what your model is doing by peering into the stateflow diagram. Debugging your algorithm time in Simulink environment is a lot easier than debugging in the C++ code. Here, we see a robot reached the stop sign successfully. Everything seems to be working as expected. What do we do to deploy the Simulink model as a ROS node running on the VM.

You go to the Apps tab in Simulink toolbar, and click Robot Operating System. Select ROS 2 as the target. This brings up a new tab called Robot. We go to this new tab and make changes in the hardware simulation configuration as needed. We then click on the Build button to generate code from our Simulink model. Here, you can see the diagnostic view is opened. And it's basically publishing the stages of the code generation. The code is generated. In the source folder, we have a complete ROS package configured to work with the ROS build system.

Now, we take this package to our ROS 2 workspace and open to VM. We open the terminal window, and build the ROS package using Colcon Builder. You can then run the standalone node without MATLAB or Simulink.

While this node is running in the VM, we can kick off a companion Simulink model for visualization. We can display what robot is seeing by receiving messages published on the image topic. We can also look at the last comments and start and stop the robot as desired.

While we built the ROS node in the VM, we could just as easily take the ROS node and load it into the actual robot. What we do in the simulation environment can be extended to the real hardware. This is the beauty of the ROS design framework.

Here's our second application example-- automated valet parking. This example has four sets of sub-examples to differentiate by the use of MATLAB with ROS or ROS 2, and Simulink with ROS or ROS 2. Parking a car autonomously is a challenging problem, and there are various ways to tackle the problem. This example demonstrates one way of realizing an automated parking system. There are several core modules required for navigation problems, such as a behavioral planner and path planner, a controller, and a vehicle model. Each one of these core modules are represented as separate nodes on a ROS network. Nodes communicate with each other by a standard publish/subscribe messaging.

For simplicity, this example focuses on planning control and vehicle modeling. We use pre-recorded data to substitute for localization information.

There are four core modules in the Simulink model-- behavioral planner, motion planner, controller, and a simplified vehicle model. There is a feedback loop from vehicle model to controller through behavioral and motion planners. Behavioral planner reads the current vehicle codes from the ROS 2 network. It computes to waypoints and checks if the vehicle has reached the final pose.

The motion planner completes a feasible path through the environment map. The desired trajectory is sent to the controller, which calculates the steering and velocity comments to move the vehicle.

The single parking valet models can be broken out into four separate models to facilitate deployment as a distributed system. Simulink signal lines going in between various modules are replaced by ROS publish and subscribe blocks. Motion planner, for example, calculates and sends the desired trajectory to the controller over the ROS network.

We can then deploy the behavioral planner, the motion planner, and the controller, as standalone ROS nodes running an actual vehicle, and replace simulation dynamics by the actual vehicle dynamics.

To wrap up, here are the key takeaways. ROS Toolbox connects MATLAB and Simulink to ROS and ROS 2. ROS Toolbox enables data analysis, desktop prototyping, and deployment workflows. You complement traditional ROS by combining it with The MathWorks tools. ROS has a rich software infrastructure and a vibrant user community. MATLAB and Simulink offer rich tooling for application-specific verticals, and have differentiator, like Stateflow and V&Vtools.

To learn more about The MathWorks as robotics tools and ROS capabilities, please visit our product web pages on that The MathWorks.com, and download a trial to check out the reference examples. We have a robotics discovery page, user stories, and videos for further information. Don't forget to visit our MATLAB and Simulink Robotics GitHub. Thank you for your attention. Don't forget to visit The MathWorks.com/robotics web page for information and resources.