UE4.26: Use Iphone ARKit for NDisplay Frustum Camera

Via sending OSC data over nDisplay to control Frustum camera | Unreal Engine 4.26.

This test been in my heart for a long time, since I wanna know if iPhone is steady enough to do camera tracking for nDisplay. So without further talk, here's the testing video below.

In the video, the frustum view does respond to my control via iPhone ARkit, though there's a delay between my movements and the frustum view. There is a known fact of nDisplay latency around 0.33 second. You can find more info on In-Camera VFX with UE4 | SIGGRAPH 2019 | Unreal Engine. In short, due to the process of rendering multiple displays and get all the data through, camera tracking data is always show a little bit behind on the LED/Screen. So for this test, the latency is normal.

Even though latency is expected. But there are 3 things I am going to point out here.

In this test, Iphone XR is connecting via WiFi. It is not very steady. It happen quite often just after few minutes using it and the tracking world origin noticeable drafting away.
Due to the protocol is using WiFi. The translate data may not go through for a second and connect later again with a big jump.
Iphone XR make world origin where you start the ARKit which could make the calibration harder only if you're making sure every time you start at the same position and you don't need to calibrate in Unreal each time you start ARKit at a different position.

Overall, I think it's not a good camera tracking device for a real shooting usage. But if you want to know how to do it, here's the way.

The logic tree of this test:

STEP 1: ARKit and OSC

I use ZIG SIM PRO for getting and sending ARKit data out of my iPhone. This app is very handy and it has almost every data you can think of like ARKit, Acceleration, Gravity, Gyro, Quaternion, Pressure, Touch, GPS and etc. For this step, I just choose ARKit as type of data that I want to send out. And setup my receiving computer IP and start.

STEP 2: OSC Receiver in Unreal Engine and Transform Data

This is a blueprint actor which the function is receiving OSC data and also applying data to a reference cine camera actor.

STEP 3: Play and Test

Add this OSC Blueprint Actor to the level, and then reference the cine camera. Again make sure your phone is using ZIG SIM PRO and start sending data.

Hit play or launch nDisplay then it should print the ARKit data on the left top corner and your camera tracking.

Last Update: June 5, 2021

Software: Unreal Engine 4.26, ZIG SIM PRO

OS: Windows 10

Specs: RTX 3070, AMD 3900x, 64GB RAM, iPhoneXR

Reference:

Unreal Engine OSC plugin - https://docs.unrealengine.com/4.27/en-US/WorkingWithAudio/OSC/
ZIG SIM PRO - https://apps.apple.com/us/app/zig-sim-pro/id1481556614