After you install the Automated Driving Toolbox™ Interface for Unreal Engine® 4 Projects support package as described in Install Support Package for Customizing Scenes, you can simulate in custom scenes simultaneously from both the Unreal® Editor and Simulink®. By using this co-simulation framework, you can add vehicles and sensors to a Simulink model and then run this simulation in your custom scene.
If you open your Unreal project file directly in the Unreal Editor, Simulink is unable to establish a connection with the editor. To establish this connection, you must open your project from a Simulink model.
Open a Simulink model configured to simulate in the Unreal Engine environment. At a minimum, the model must contain a Simulation 3D Scene Configuration block. For example, open a simple model that simulates a vehicle driving on a straight highway. This model is used in the Design Lane Marker Detector Using Unreal Engine Simulation Environment example.
In the Simulation 3D Scene Configuration block of this model,
set the Scene source parameter to
In the Project parameter, browse for the project file that contains the scenes that you want to customize.
For example, this sample path specifies the
project that comes installed with the Automated Driving Toolbox Interface for Unreal Engine 4 Projects support
Click Open Unreal Editor. The Unreal Editor opens and loads a scene from your project.
The first time that you open the Unreal Editor from Simulink, you might be asked to rebuild
UE4Editor DLL files or
AutoVrtlEnv module. Click Yes to rebuild
these files or modules. The editor also prompts you that new plugins are available.
Click Manage Plugins and verify that the MathWorks
Interface plugin is installed. This plugin is the
MathWorksSimulation.uplugin file that you copied into your Unreal
Editor installation in Install Support Package for Customizing Scenes.
When the editor opens, you can ignore any warning messages about files with the name
'_BuiltData' that failed to load.
If you receive a warning that the lighting needs to be rebuilt, from the toolbar above the editor window, select Build > Build Lighting Only. The editor issues this warning the first time you open a scene or when you add new elements to a scene.
If you are using a scene from the
RRScene project that is part of the Automated Driving Toolbox Interface for Unreal Engine 4 Projects support package, skip this section. However, if you create a new scene
based off of one of the scenes in this project, then you must complete this
The first time that you open a custom scene from Simulink, you need to associate, or reparent, this project with the Sim3dLevelScriptActor level blueprint used in Automated Driving Toolbox. The level blueprint controls how objects interact with the Unreal Engine environment once they are placed in it. Simulink returns an error at the start of simulation if the project is not reparented. You must reparent each scene in a custom project separately.
To reparent the level blueprint, follow these steps:
In the Unreal Editor toolbar, select Blueprints > Open Level Blueprint.
In the Level Blueprint window, select File > Reparent Blueprint.
Click the Sim3dLevelScriptActor blueprint. If you do
not see the Sim3dLevelScriptActor blueprint listed, use
these steps to check that you have the
MathWorksSimulation plugin installed and
In the Unreal Editor toolbar, select Settings > Plugins.
In the Plugins window, verify that the MathWorks Interface plugin is listed in the installed window. If the plugin is not already enabled, select the Enabled check box.
If you do not see the MathWorks Interface plugin in this window, repeat the steps under Copy MathWorksSimulation Plugin to Unreal Editor and reopen the editor from Simulink.
Close the editor and reopen it from Simulink.
Close the Level Blueprint window.
After you open the editor from Simulink, you can modify the scenes in your project or create new scenes.
In the Unreal Editor, scenes within a project are referred to as levels. Levels come in several types, and scenes have a level type of map.
To open a prebuilt scene from the
RRScene.uproject file, in the Content
Browser pane below the editor window, navigate to the Content > Maps folder. Then, select the map that corresponds to the scene you
want to modify.
This table shows the map names in the
project as they appear in the Unreal Editor. It also shows their corresponding scene names as they
appear in the Scene name parameter of the
Simulation 3D Scene Configuration block.
|Unreal Editor Map||Automated Driving Toolbox Scene|
|Double Lane Change|
|Large Parking Lot|
|US City Block|
AutoVrtlEnv.uproject file does not include
the Virtual Mcity scene.
RRScene project contains only one scene:
RRHighway. This scene is used in the Highway Lane Following with RoadRunner Scene
example and is not selectable from the Scene name
parameter of the Simulation 3D Scene Configuration
To open a scene within your own project, in the Content Browser pane, navigate to the folder that contains your scenes.
To create a new scene in your project, from the top-left menu of the editor, select File > New Level.
Alternatively, you can create a new scene from an existing one. This technique is
useful, for example, if you want to use one of the prebuilt scenes in the
AutoVtrlEnv project as a starting point for creating your own
scene. To save a version of the currently opened scene to your project, from the
top-left menu of the editor, select File > Save Current As. The new scene is saved to the same location as the existing
In your scenes, you can specify trajectory waypoints that the vehicles in your scene can follow.
If your scene is based off one of the prebuilt scenes in the
AutoVrtlEnv project, then specify waypoints using
the process described in the Select Waypoints for Unreal Engine Simulation
example. This example shows how to interactively draw waypoints on 2-D
top-down maps of the prebuilt scenes.
If your is not based off of a prebuilt scene, then before using the Select Waypoints for Unreal Engine Simulation example, you must first generate a map of your scene. See Create Top-Down Map of Unreal Engine Scene.
In the Unreal Editor, elements within a scene are referred to as assets. To add assets to a scene, you can browse or search for them in the Content Browser pane at the bottom and drag them into the editor window.
When adding assets to a scene that is in the
project, you can choose from a library of driving-related assets. These assets are
built as static meshes and begin with the prefix
SM_. Search for these objects in the Content
For example, add a stop sign to a scene in the
In the Content Browser pane at the bottom of the editor, navigate to the Content folder.
In the search bar, search for
the stop sign from the Content Browser into the
editing window. You can then change the position of the stop sign in the
editing window or on the Details pane on the right,
in the Transform section.
The Unreal Editor uses a left-hand Z-up coordinate system, where the Y-axis points to the right. Automated Driving Toolbox uses a right-hand Z-up coordinate system, where the Y-axis points to the left. When positioning objects in a scene, keep this coordinate system difference in mind. In the two coordinate systems, the positive and negative signs for the Y-axis and pitch angle values are reversed.
For more information on modifying scenes and adding assets, see Unreal Engine 4 Documentation.
To migrate assets from the
AutoVrtlEnv scene into the
RRScene project or to your own project, see Migrating Assets in the Unreal Engine documentation.
To obtain semantic segmentation data from a scene, then you must apply stencil IDs to the objects added to a scene. For more information, see Apply Semantic Segmentation Labels to Custom Scenes.
Verify that the Simulink model and Unreal Editor are configured to co-simulate by running a test simulation.
In the Simulink model, click Run.
Because the source of the scenes is the project opened in the Unreal Editor, the simulation does not start. Instead, you must start the simulation from the editor.
Verify that the Diagnostic Viewer window in Simulink displays this message:
In the Simulation 3D Scene Configuration block, you set the scene
source to 'Unreal Editor'. In Unreal Editor, select 'Play' to view the
This message confirms that Simulink has instantiated vehicles and other objects in the Unreal Engine environment.
In the Unreal Editor, click Play. The simulation runs in the scene currently open in the Unreal Editor.
If your Simulink model contains vehicles, these vehicles drive through the scene that is open in the editor.
If your Simulink model includes sensors, these sensors capture data from the scene that is open in the editor.
To control the view of the scene during simulation, in the Simulation 3D Scene Configuration block, select the vehicle name from the Scene view parameter. To change the scene view as the simulation runs, use the numeric keypad in the editor. The table shows the position of the camera displaying the scene, relative to the vehicle selected in the Scene view parameter.
To restart a simulation, click Run in the Simulink model, wait until the Diagnostic Viewer displays the confirmation message, and then click Play in the editor. If you click Play before starting the simulation in your model, the connection between Simulink and the Unreal Editor is not established, and the editor displays an empty scene.
If you are co-simulating a custom project, to enable the numeric keypad, copy the
DefaultInput.ini file from the support package installation
folder to your custom project folder. For example, copy
After tuning your custom scene based on simulation results, you can then package the scene into an executable. For more details, see Package Custom Scenes into Executable.