Virtual and augmented reality live or die on frame rate. Miss the headset's refresh and the wearer gets sick, so every millisecond you spend drawing the left eye is a millisecond you do not have for the right. We have been putting real 3D engines, not canned demos, into headsets since the Oculus DK2, and underneath the changing hardware the problem stays the same: draw a large, real scene twice or more, fast enough that nobody notices the machine straining.
At I/ITSEC 2015 we demonstrated OpenSceneGraph and osgEarth running on an early Oculus DK2, real terrain and a real scene graph in a headset at a time when most VR was still spinning cubes.
Rendering two eyes without dropping frames
A stereo headset renders the scene at least twice, once per eye, and modern devices push that further with more views and higher refresh rates. Doing that the naive way, one eye after the other, wastes the traversal. We wrote a Multiview extension for OpenSceneGraph that renders the views in parallel: the scene graph is traversed once and the eyes are drawn together. On headsets with eye tracking we have done foveated rendering, spending full resolution only where the eye is actually looking and dropping detail in the periphery, which is where the frame-time budget on a device like the Varjo actually comes from.
(Some of the) Headsets we have worked on
Oculus DK2 through the consumer Rift, the HTC Vive (we worked on the MakeVR port to it), Varjo's high-resolution eye-tracked headsets, and a defense program on the Meta Quest 2 and Quest Pro, plus the Microsoft HoloLens 2, which we reached by porting OpenSceneGraph to it through ANGLE for the U.S. Army's IVAS program. The device changes every couple of years. The job of feeding it a real scene at full rate does not.
Virtual data in, not just pixels out
XR is an input problem as much as a display problem, and our history there runs back to the 1990s, when we built a serial interface for the Nintendo Power Glove using a PIC microcontroller. Since then we have integrated glove and spatial controllers with OpenSceneGraph, including the P5 Data Glove and the Razer Hydra, so a tracked hand or wand drives the same scene graph that renders the view.
Â
Augmented reality in the real world
We worked on a maritime augmented reality trainer for Rockwell Collins, a land-based system that uses AR headsets and greenscreen to overlay navigation and situational data the way a crew would see it at sea, over top of a fully functional simulated tangible control console.
Mobile, autostereo, and holographic
Our XR work goes back to phone-based viewers: Google Cardboard and the viewmaster-style holders that turned a smartphone into a stereo display, including the VR app we built to present a special VR movie tied to the film Allegiant (our part was the app, not the film). We have also gone past the headset entirely. We have built custom multiheaded rendering farms, banks of multi-output graphics cards, to drive horizontal-segmented, lenticular-style autostereoscopic and holographic displays, and we have worked with glasses-free devices such as the Leia Lumepad tablet.
How it connects
XR is where several of our disciplines meet. The engine and the Multiview rendering are real-time 3D graphics. The training and mission content is visual simulation. The tracking that keeps augmented reality glued to the world is computer vision. One team that has built all of it is worth more than three vendors who each own a slice.
Who works on it
AlphaPixel is a US-owned small business, founded in 2004, with senior developers who each have 25-plus years in real-time 3D. DLA DD2345 / ITAR registered. We take closed-source defense work and open source alike, and we do not sell a headset, so our advice is not a sales pitch for one.
Talk to us
Tell us what you’re building and where it’s stuck. We’ll tell you straight whether it’s something we can help with, and how we’d approach it.
Or see how we run consulting and software development engagements.
