Back to the Main page
The OptiTrack Unreal Engine 4 Plugin allows real-time streaming of live rigid body tracking data from Motive into Unreal Engine 4 (abbreviated as UE4). The plugin also features HMD tracking integration for virtual reality applications. Once the plugins are added to the Unreal Engine directory, OptiTrack – NatNet Streaming Client and OptiTrack – Oculus Rift plugins will be available within the UE4 project. This article focuses on the streaming workflow as well as setting up the HMD in UE4 using the plugin. For general instructions on setting up the system, please refer to the Getting Started guide.
The NatNet Streaming Client plugin will allow you to stream rigid body tracking data from Motive onto Unreal Engine, and the Oculus Rift plugin will allow you to implement HMD tracking onto your scene. Once the plugins are enabled, OptiTrack Client Origin object can be called, and it connects the project to the server application (i.e. Motive) via NatNet protocol. Position and orientation of the Client Origin object reconciles the global origin in Motive, and all streamed tracking data will be referenced against to it. This origin can be placed anywhere in the scene and scaled to different sizes. Once the project is connected to the server application, imported tracking data can be used to instantiate the HMD and animate the objects in virtual reality applications.
Virtual Reality Experience
OptiTrack motion capture systems provide motion tracking with outstanding accuracy, and all of which can be accomplished with extremely low latency. Utilizing its tracking capability, an OptiTrack system can be used to provide fully immersive virtual reality experience by integrating head-mounted display (HMD) device. From this combined setup, even tiny movements can be precisely captured and reflected in the virtual scene.
As of OptiTrack Unreal Engine 4 Plugin 1.3.0, Oculus DK2, CV1, and Gear VR HMD models are supported
To stream the rigid body data, follow the general data streaming workflow provided from the Data Streaming page. Make sure appropriate network interface is defined and the Stream Rigid Bodies is enabled under the streaming settings.
When streaming the data over a wifi network, Unicast transmission must be used.
This section includes instructions on installing and configuring the Unreal Engine plugin to receive tracking data from Motive.
C:\Program Files\Epic Games\4.##\Engine\Plugins
Note: When using the Oculus Plugin, make sure to disable the built-in Oculus Rift (or the Gear VR) and Stream VR HMD plugins under virtual reality plugins category. Otherwise, the editor may crash, or they may end up having a higher priority than the OptiTrack Oculus Plugin, and if enabled, will be used in preference to it.
The OptiTrack Client Origin enables the Unreal Engine (client) to communicate with the NatNet server where Motive’s rigid body and HMD tracking data is obtained. In the client object, both the communication address and the rigid body ID for the HMD are configured. Add this object simply dragging and dropping the OptiTrack Client Origin from the Modes panel into the scene.
Once the OptiTrack Client Origin is placed, position and orientation of this object will reconcile the global origin of Motive in Unreal Engine. In other words, the tracking data will be represented according to where this Client Origin object is placed and how it is oriented.
Global Origin: Both position and orientation of the OptiTrackClientOrigin will represent the global origin of the tracking volume within Motive.
Advance settings: Auto-initialize
Starting from the plugin version 1.9.2, the skeleton component has been added to allow streaming of the skeleton via bone mapping, not retargetting. This means that the bone segments in Motive can be mapped directly to the character model (bone mapping), instead of being translated into something that is usable by a more abstract biped model (retargeting). The instructions for setting up the skeleton streaming component is documented in the following page:
Go to: Unreal Engine Skeleton Streaming
Integrating skeleton tracking in Unreal Engine requires a 3D retargeting process. This can be accomplished through a third party software that retargets the 3D data to produce realistic motion in a scene. For more information, contact us.
Notes for IKinema LiveAction for UE 4.18.
C:\Program Files\Epic Games\UE_4.18\Engine\Plugins\UE_4.18\OptiTrack\OptitrackNatNet\ThirdParty\NatNetSDK\lib
C:\Program Files\Epic Games\UE_4.18\Engine\Plugins\LiveAction\Source\ThirdParty\NatNet\NatNetSDK\lib\
First of all, make sure the OptiTrack – OculusVR is enabled in your project in the plugins panel in order to integrate the HMD. Once the Client Origin object is called into the project, input the streaming ID for the HMD rigid body to the HMD Rigid Body ID section under the OptiTrack Client Origin. Then within the project World Settings, set the Default Pawn Class to be the OptiTrackHMDPawn class. Once this is set up, follow the HMD setup steps in the next section to prepare your HMD object in Motive.
Gear VR devices can also be integrated and the tracking information can be streamed via wifi connections using a router with sufficient bandwidth. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unreal Engine, refer to UE 4 documentations.
Setup and optimize the motion capture volume as explained in the Getting Started guide or the Hardware Setup documentations. If you plan to install any obstacles (e.g. walls) within the capture volume, make sure they are non-reflective, and place and orient the cameras so that every corner is thoroughly captured by multiple cameras. For typical rigid body tracking, attach the rigid body markers as instructed from the Rigid Body Tracking page.
General Setup Steps
With the HMD bracket, simply attach the active marker bracket onto an CV1 HMD and use the active markers on it to define a rigid body.
When using the OptiTrack Active Marker Kit, active markers are actively labeled individually from their own unique illumination patterns, and only the matching active markers get contributed to the rigid body solve. The biggest benefit from the individual labeling is that markers can be placed in perfectly symmetrical placements among multiple rigid body assets.
When attaching retroreflective markers, make sure markers are securely attached and readily captured by the cameras. For attaching the markers, we recommend using our 20 mm wide and 30 mm tall M4 threaded plastic marker bases with Acrylic adhesives, available at the webstore, to attach the markers onto the HMD.
A markered HMD will be defined as a rigid body in Motive. When placing markers, make sure the placement asymmetry is respected in the arrangement within the HMD. Also, the marker arrangements between multiple HMDs must be incongruent. For more details, read about marker placement from the Rigid Body Tracking page.
In Motive, select the markers that are attached on the HMD and create a rigid body (Hotkey: CTRL + T) as you would do for other rigid bodies. Once the rigid body is created, select the asset and open the Properties pane. The following settings need to be configured for HMD tracking.
Deflection setting is the tolerable distance, in millimeters, that a rigid body marker may deviate from its expected position before it is unlabeled and unassociated with the rigid body. The deflection is set to 4 mm by default. For HMD tracking applications, we recommended lowering this value to 3 or 2. This will reduce the amount of computation required for labeling, and overall latency may be reduced.
The Steaming ID of a rigid body is a number that can be used for referencing the specific rigid body in external applications. Log this value for each rigid body (including the HMD(s)) in the scene. This number will be used to associate the assets in the client applications using the plugin devices.
This setting may need to be modified when tracking HMDs with passive markers. To prevent the swapping of the rigid body definition, set this setting to 4. When tracking multiple HMDs, there could be limitations to a variety of unique marker arrangements that could be achieved. If this value is set to a lower value, a set of three markers on an HMD may be congruent to another set in a different HMD, and the rigid body definitions may be switched in Motive.
After the rigid body definition have been created for the HMD, the position and orientation of the rigid body pivot point need to be carefully adjusted for HMD tracking:
In general, you will be utilizing the HMD calibration tool for precisely positioning and orienting the HMD rigid body. However, if, for some reason, you want to manually position the pivot point without using the HMD Calibration feature, follow the instruction on this page: Manually Calibrating the HMD Pivot Point.
Rigid body pivot can be easily calibrated using the HMD Calibration tool. Supported only for Oculus HMD models only. Does not work with GearVR.
You can also use the HMD Calibration feature to easily calibrate the pivot point of the active marker HMD clips.
Calibration of HMDs with passive markers utilizes both the OptiTrack mocap system and the Oculus Tracker for precisely calculating the location of the pivot point of an Oculus HMD. When using this feature, both systems must be tracking the HMD rigid body.
Tip: Once you have the rigid body asset for the HMD configured, you can export the asset into a TRA file for future uses. Importing the TRA file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement must remain unchanged in order to re-load previously created rigid bodies.
Once you have connected the client object and configured the script components on the assets from the scene. Play the scene, and if the properties are correctly configured, the assets in the game scene will be animated according to the position and orientation of the rigid bodies and the HMD in Motive. When playing the scene, the HMD and the assigned actor objects will be placed in respect to their location in Motive’s coordinate system disregards to their position in the editor.
When setting up multiplayer games on wireless clients, it is more beneficial for each client to make direct connection to both the tracking-server (Motive) and the game-server, rather than rebroadcasting the streamed tracking data through the game-server. Then, any of the game related actions that interacts with the tracking data can be processed on the game-server, and this server can send out the corresponding updates to the wireless clients. This allows the wireless clients to only receive both the tracking data or updates without having to send back any information; in other words, minimizing the number of data transfers needed. If wireless clients are sending data there will be a minimum of two transfers on the wireless network, and each transfer of data through wireless network is at risk of latency or lost packets.