OptiTrack Unreal Engine 4 Plugin
The OptiTrack Unreal Engine 4 Plugin allows real-time streaming of live rigid body tracking data from Motive into Unreal Engine 4 (abbreviated as UE4). The plugin also features HMD tracking integration for virtual reality applications. Once the plugins are added to the Unreal Engine directory, OptiTrack – NatNet Streaming Client and OptiTrack – Oculus Rift plugins will be available within the UE4 project. The NatNet Streaming Client plugin will allow you to stream rigid body tracking data from Motive onto Unreal Engine, and the Oculus Rift plugin will allow you to implement HMD tracking onto your scene. Once the plugins are enabled, OptiTrack Client Origin object can be called. The Client Origin object connects the project to the server application via NatNet protocol and imports the tracking data. When the Client Origin object is imported into the scene, it reconciles the global origin in Motive, and all streamed tracking data will be referenced against to it. This origin can be positioned anywhere in the scene and scaled to different sizes. When the project is connected to the server application, imported tracking data can be used to instantiate the HMD and animate the objects in virtual reality applications.
Virtual Reality Experience
OptiTrack motion capture systems provide object tracking and full-body motion tracking with outstanding accuracy, and all of which can be accomplished with extremely low latency. Utilizing its tracking capability, an OptiTrack system can be used to provide fully immersive virtual reality experience by integrating head-mounted display (HMD) device. From this combined setup, even tiny movements can be precisely captured and reflected in the virtual space providing the most immersive virtual reality experience. The OptiTrack Unreal Engine 4 Plugin allows you to stream real-time rigid body and HMD tracking data into UE4 using Motive. This article focuses on the streaming workflow using the plugin. For general instructions on setting up the system, please refer to the Getting Started guide.
|Plugin Version||UE Version||Oculus SDK|
As of OptiTrack Unreal Engine 4 Plugin 1.3.0, Oculus DK2, CV1, and Gear VR HMD models are supported
- 1 Motive Data Streaming Setup (Server)
- 2 Unreal Engine Setup (Client)
- 3 HMD Tracking Setup
Motive Data Streaming Setup (Server)
To stream the rigid body data, follow the general data streaming workflow provided from the Data Streaming page. Make sure appropriate network interface is defined and the Stream Rigid Bodies is enabled under the streaming settings.
Streaming in Motive
- Open the Data Streaming Pane in Motive
- Select the network interface address for streaming data.
- Select desired data types to stream under streaming options.
- When streaming skeletons, set the appropriate bone naming convention for client application.
- Check Broadcast Frame Data at the top.
- Configure streaming settings and designate the corresponding IP address from client applications
- Stream live or playback captures
When streaming the data over a wifi network, Unicast transmission must be used.
Unreal Engine Setup (Client)
This section includes instructions on installing and configuring the Unreal Engine plugin to receive tracking data from Motive.
Setting Up the Plugin
- Download the plugin ZIP file. (Wait until this is publically available).
- Extract the contents from the ZIP file.
- Open the OptiTrack folder, transfer the OptiTrackNatNet and OptiTrackOculusRift folders into the Unreal Engine plugin directory. By default, this directory is installed in the
C:\Program Files\Epic Games\4.##\Engine\Pluginsfolder. There will be other plugins installed in the folder already. Copy and paste the OptiTrack plugin folders into this directory.
- Open/Create a UE4 project, and under the plugin devices, enable both of the OptiTrack plugin devices.
- Under the Edit menu on the main command bar, open the plugin panel.
- Browse to Virtual Reality section and enable the OptiTrack plugins (OptiTrack - NatNet Streaming Client and OptiTrack - Oculus Rift) that are added to the folder.
Note: When using the Oculus Plugin, disable the built-in Oculus Rift (or the Gear VR) and Stream VR HMD plugins under virtual reality plugins category, as they will by default have a higher priority than the OptiTrack Oculus Plugin, and if enabled, will be used in preference to it.
Unreal Engine Client Setup
OptiTrack Client Origin
The OptiTrack Client Origin enables the Unreal Engine (client) to communicate with the NatNet server where Motive’s rigid body and HMD tracking data is obtained. In this object, both the communication address and the rigid body ID for the HMD is configured. To add this object, simply drag and drop the OptiTrack Client Origin from the Modes panel into the scene.
Global Origin: Location of this client object will reconcile the global origin of Motive in Unreal Engine. In other words, the tracking data will be represented according to where this Client Origin object is placed in the scene.
Connecting Unreal Engine to Motive
- Once the plugin is properly added and enabled, OptiTrack Client Origin class will be available from the Modes panel in UE.
- Drag and drop the OptiTrack Client Origin into the scene.
- Select the instantiated OptiTrackClientOrigin object from the World Outliner panel.
- Under the Details panel, expand the OptiTrack Client Origin properties.
- Enter matching server and client network address information that was defined from the Data Streaming Pane in Motive.
- In Motive, make sure the Broadcast Frame Data entry is checked.
- Now the UE4 project is connected with Motive, and the tracking data will be available in UE4 when the scene is played.
Animating Rigid Body
Actor objects in Unreal Engine can be animated using rigid body tracking data from Motive. This can be used to animate props in the scene. To do this, you will need to create a rigid body asset in Motive, and it must be linked with the object in Unreal Engine using the same Streaming ID.
- Select an object in the scene.
- Under the details panel, use the +Add Component button to browse and associate the OptiTrack: OptiTrack Rigid Body component with the object.
- Under the OptiTrack Client Origin component properties, match the Tracking ID with Streaming ID of the rigid body asset that was defined in Motive.
Integrating skeleton tracking in Unreal Engine requires a 3D retargeting process. This can be accomplished through a third party software that retargets the 3D data to produce realistic motion in a scene. For more information, contact us.
Integrating HMD (Oculus DK2, CV1)
After the Client Origin object is called into the project, the HMD can be integrated into Unreal Engine. First of all, make sure the OptiTrack – Oculus Rift plugin is enabled in your project. In order to instantiate HMD object tracking in UE, you will need to input the Streaming ID for the HMD, and use a Game Mode with the OptiTrackHMDPawn class defined as the default pawn class.
When integrating the HMD, make sure to follow the HMD setup steps and prepare your HMD object in Motive.
Linking the HMD Tracking Data
- Enable the OptiTrack – Oculus Rift plugin from the plugins panel.
- Open the World Settings panel under the Window tab.
- If not done already, create a new GameMode by adding a new blueprint under the Game Mode section.
- Change the Default Pawn Class to OptiTrackHmdPawn. You may need to restart the editor before this change takes effect.
- In UE, select the OptitrackClientOrigin and access the Details panel.
- Expand the OptiTrack Client Origin section.
- In the HMD Rigid Body ID entry, input the respective Streaming ID value configured from the rigid body properties in Motive.
- Now the tracking data is linked to the Unreal Engine project, and the HMD will display the scene when it is played.
Integrating HMD (Android: GearVR)
Gear VR devices can also be integrated and the tracking information can be streamed via wifi connections using a router with sufficient bandwidth. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unreal Engine, refer to UE 4 documentations.
- [UE 4] For developing Android applications in Unreal Engine, make sure the project environment is set up for Android development: Android Quick Start Guide .
- [UE4] Follow the GearVR Quick Start Guide and configure the project for GearVR development.
- [UE4] Generate and include the OSIG file (oculus signature) in the directory. See: Oculus Signature File Generator.
- [Motive] Under the Data Streaming pane, set the Network Type setting under the Advanced Settings to Unicast. Note that the plugin currently only supports the Unicast broadcasting for streaming onto Android, multicasting will be supported in the future releases.
Connecting via Wifi
- 1. [Android] Connect the smartphone to the internet router which the host PC (server) is connected to.
- 2. [UE4] Configure the Client object.
- - Use Connection Type: Unicast.
- - Set the Local IP address to (0.0.0.0). The local address is not required for unicast connections.
- - For the Server Address, connect to the IP address that the host PC (server) is connected to. This could also be found from the Android OS when the smartphone is connected to the wifi router.
- 3. [UE4] Connect a smartphone into the PC, build and run the project.
- 4. Make sure Motive is streaming, and the Gear VR HMD tracking data will be streamed into the UE4 application.
Once you have connected the client object and configured the script components on the assets from the scene. Play the scene, and if the properties are properly configured, the assets in the UE scene will be animated according to the position and orientation of the rigid bodies and the HMD in Motive. When playing the scene, the HMD and the assigned actor objects will be placed in respect to their location in Motive’s coordinate system disregards to their position in the UE4 editor.
HMD Tracking Setup
Setup and optimize the motion capture volume as explained in the Getting Started guide or the Hardware Setup documentations. If you plan to install any obstacles (e.g. walls) within the capture volume, make sure they are non-reflective, and place and orient the cameras so that every corner is thoroughly captured by multiple cameras. For typical rigid body tracking, attach the rigid body markers as instructed from the Rigid Body Tracking page.
General Setup Steps
- Attach the markers on the HMD
- Create a rigid body asset
- Calibrate the Pivot Point of the rigid body
- Configure the rigid body settings
When attaching retroreflective markers, make sure markers are securely attached and readily captured by the cameras. For attaching the markers, we recommend using our 20 mm wide and 30 mm tall M4 threaded plastic marker bases with Acrylic adhesives, available at the webstore, to attach the markers onto the HMD.
A markered HMD will be defined as a rigid body in Motive. When placing markers, make sure the placement asymmetry is respected in the arrangement within the HMD. Also, the marker arrangements between multiple HMDs must be incongruent. For more details, read about marker placement from the Rigid Body Tracking page.
Attachment & Placement
With the HMD bracket, simply attach the active marker bracket onto an CV1 HMD and use the active markers on it to define a rigid body.
When using the OptiTrack Active Marker Kit, active markers are actively labeled individually from their own unique illumination patterns, and only the matching active markers get contributed to the rigid body solve. The biggest benefit from the individual labeling is that markers can be placed in perfectly symmetrical placements among multiple rigid body assets.
Oculus Positional Tracker (Rift sensors)
- When using an OptiTrack motion capture system to provide a solution for tracking Oculus HMD, avoid connecting its positional tracker, the Rift sensor, to the host PC.
- When the tracker is connected, IR LEDs on the connected HMD will illuminate, and it could interfere negatively with the IR tracking of the motion capture system.
- When Oculus tracker is not recognized by the computer, an error message may appear but you should be able to ignore the error message (Critical Error - can't find the sensor) and proceed without the tracker.
Create an HMD Rigid Body
In Motive, select the markers that are attached on the HMD and create a rigid body (Hotkey: CTRL + T) as you would do for other rigid bodies. Once the rigid body is created, select the asset and open the Properties pane. The following settings need to be configured for HMD tracking.
Deflection setting is the tolerable distance, in millimeters, that a rigid body marker may deviate from its expected position before it is unlabeled and unassociated with the rigid body. The deflection is set to 4 mm by default. For HMD tracking applications, we recommended lowering this value to 3 or 2. This will reduce the amount of computation required for labeling, and overall latency may be reduced.
Use the ray-based tracking algorithm. The tracking algorithm setting determines which protocol to use for solving the rigid body in Motive. This algorithm is recommended because it allows tracking to be less susceptible to marker occlusions and solves rigid bodies more robustly without introducing trajectory gaps. This is more suitable for tracking the HMDs since even minor occlusions can acutely affect the experience.
The User Data value for each rigid body is used as an identification number to reference the rigid body in external applications. Log this value for each rigid body (including the HMD(s)) in the scene. This number will be used to associate the assets in the client applications using the plugin devices.
Min Marker Count
This setting may need to be modified when tracking HMDs with passive markers. To prevent the swapping of the rigid body definition, set this setting to 4. When tracking multiple HMDs, there could be limitations to a variety of unique marker arrangements that could be achieved. If this value is set to a lower value, a set of three markers on an HMD may be congruent to another set in a different HMD, and the rigid body definitions may be switched in Motive.
Calibrate Rigid Body Pivot Point
After the rigid body definition have been created for the HMD, the position and orientation of the rigid body pivot point need to be carefully adjusted for HMD tracking:
- The pivot point is placed on the bridge of the nose; specifically, right at the midpoint between two eyes.
- Orientation axes of the rigid body should precisely align with the physical HMD component.
Manually Calibrating the HMD Pivot Point
In general, you will be utilizing the HMD calibration tool for precisely positioning and orienting the HMD rigid body. However, if, for some reason, you want to manually position the pivot point without using the HMD Calibration feature, follow the instruction on this page: Manually Calibrating the HMD Pivot Point.
HMD Calibration Tool
Rigid body pivot can be easily calibrated using the HMD Calibration tool. Supported only for Oculus HMD models only. Does not work with GearVR.
Using the HMD Calibration Tool (Passive Markers)
Calibration of HMDs with passive markers utilizes both the OptiTrack mocap system and the Oculus Tracker for precisely calculating the location of the pivot point of an Oculus HMD. When using this feature, both systems must be tracking the HMD rigid body.
- Connect the Oculus Tracker and have the Oculus software running in the background.
- Launch Motive.
- [Motive] Create a rigid body asset from the markers on the oculus HMD. Read: Creating Rigid Body.
- [Motive] Open the Trackables pane. Tools → Trackables → HMD Calibration tab
- [Motive] Select the HMD rigid body. Under Detected Type, it should indicate Custom RigidBody.
- [Motive → Trackables pane] Set the desired number of sample counts (Default: 750) and the desired orientation.
- [Motive → Trackables pane] Press start to initiate the sampling process.
- [Motive → Trackables pane] During calibration, make sure both the mocap system and the Oculus Tracker is tracking the HMD.
- [Motive → Trackables pane] Once it starts collecting the samples, point the HMD towards the Tracker and slowly rotate it and sample different orientations.
- [Motive → Trackables pane] When it finishes collecting samples, it will display the offset distance, in mm, between pivot point position detected by oculus and the motion capture system.
- [Motive → Trackables pane] If the results are satisfying, press Apply to calibrate the HMD rigid body.
Using the HMD Calibration Tool (Active HMD Clips)
You can also use the HMD Calibration feature to easily calibrate the pivot point of the active marker HMD clips.
- First of all, make sure Motive is configured for active marker tracking.
- In Live mode, select the active markers on the HMD clip and define a rigid body.
- Under Tools tab open the Trackables pane and access the HMD calibration tab.
- Hold the selected rigid body at the center of the capture volume so that as many cameras as possible can clearly capture the markers on the HMD clip.
- Select the active HMD clip rigid body in Motive.
- Make sure Detected Type shows up as OptiTrack Active HMD clip.
- Under the Samples Counts entry, assign a number of sample frames that you wish to capture for calibration
- Select the desired orientation of the rigid body. You can select from +Z forward (Unity), +X forward (Unreal), or -Z forward (Native)
- Press Start to start the calibration process.
- Slowly rotate the HMD to collect samples in diverse orientations.
- Once all necessary samples are collected, it will report the calibration result.
- If everything looks good, press Apply Calibration, and the HMD rigid body pivot point will be calibrated to given orientation.
Export the Rigid Body
Tip: Once you have the rigid body asset for the HMD configured, you can export the asset into a TRA file for future uses. Importing the TRA file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement must remain unchanged in order to re-load previously created rigid bodies.